av9ash/GitBugs huggingface dataset:
Bug: Incorrect attribute nullability after RewriteCorrelatedScalarSubquery leads to incorrect query results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/23 00:14
Affects Version/s: 3.4.0

The following query:

 
{code:java}
select * from (
 select t1.id c1, (
  select t2.id c from range (1, 2) t2
  where t1.id = t2.id  ) c2
 from range (1, 3) t1 ) t
where t.c2 is not null
-- !query schema
struct<c1:bigint,c2:bigint>
-- !query output
1	1
2	NULL
 {code}
 
should return 1 row, because the second row is supposed to be removed by IsNotNull predicate. However, due to a wrong nullability propagation after subquery decorrelation, the output of the subquery is declared as not-nullable (incorrectly), so the predicate is constant folded into True.
Bug: Sequence expression can overflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/May/23 03:04
Affects Version/s: 3.5.0

Spark has a (long-standing) overflow bug in the {{sequence}} expression.

 

Consider the following operations:

{{spark.sql("CREATE TABLE foo (l LONG);")}}
{{spark.sql(s"INSERT INTO foo VALUES (${Long.MaxValue});")}}
{{spark.sql("SELECT sequence(0, l) FROM foo;").collect()}}

 

The result of these operations will be:

{{Array[org.apache.spark.sql.Row] = Array([WrappedArray()])}}

an unintended consequence of overflow.

 

The sequence is applied to values {{0}} and {{Long.MaxValue}} with a step size of {{1}} which uses a length computation defined [here|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3451]. In this calculation, with {{{}start = 0{}}}, {{{}stop = Long.MaxValue{}}}, and {{{}step = 1{}}}, the calculated {{len}} overflows to {{{}Long.MinValue{}}}. The computation, in binary looks like:

{{{{0111111111111111111111111111111111111111111111111111111111111111 -}}}}

{{{{0000000000000000000000000000000000000000000000000000000000000000}}}}

{{{{------------------------------------------------------------------      0111111111111111111111111111111111111111111111111111111111111111 /}}}}

{{{{0000000000000000000000000000000000000000000000000000000000000001}}}}

{{{{------------------------------------------------------------------                0111111111111111111111111111111111111111111111111111111111111111 +}}}}

{{{{0000000000000000000000000000000000000000000000000000000000000001}}}}

{{{{------------------------------------------------------------------      1000000000000000000000000000000000000000000000000000000000000000}}}}

The following [check|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3454] passes as the negative {{Long.MinValue}} is still {{{}<= MAX_ROUNDED_ARRAY_LENGTH{}}}. The following cast to {{toInt}} uses this representation and [truncates the upper bits|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3457] resulting in an empty length of 0.

Other overflows are similarly problematic.
Bug: Spark Connect Session & Query lifecycle
Status: Open | Priority: Major | Resolution: 
Created: 23/May/23 15:49
Affects Version/s: 3.5.0

Currently, queries in Spark Connect are executed within the RPC handler.

We want to detach the RPC interface from actual sessions and execution, so that we can make the interface more flexible
 * maintain long running sessions, independent of unbroken GRPC channel
 * be able to cancel queries
 * have different interfaces to query results than push from server
Bug: Fix Avro data type conversion issues to avoid producing incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/23 23:01
Affects Version/s: 3.4.0

We found the following issues with open-source Avro:
 * Interval types can be read as date or timestamp types that would lead to wildly different results
 * Decimal types can be read with lower precision, that leads to data being read as {{null}} instead of suggesting that a wider decimal format should be provided
Bug: RewriteCorrelatedScalarSubquery should handle duplicate attributes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/23 18:17
Affects Version/s: 3.4.0

This is a correctness problem caused by the fact that the decorrelation rule does not dedup join attributes properly. This leads to the join on (c1 = c1), which is simplified to True and the join becomes a cross product.

 

Example query:

 
{code:java}
create view t(c1, c2) as values (0, 1), (0, 2), (1, 2)

select c1, c2, (select count(*) cnt from t t2 where t1.c1 = t2.c1 having cnt = 0) from t t1
-- Correct answer: [(0, 1, null), (0, 2, null), (1, 2, null)]
+---+---+------------------+
|c1 |c2 |scalarsubquery(c1)|
+---+---+------------------+
|0  |1  |null              |
|0  |1  |null              |
|0  |2  |null              |
|0  |2  |null              |
|1  |2  |null              |
|1  |2  |null              |
+---+---+------------------+ {code}
Bug: In expression not compatible with EqualTo Expression
Status: Open | Priority: Major | Resolution: 
Created: 13/May/23 05:11
Affects Version/s: 3.3.1

The query results of Spark SQL 3.1.1  and Hive SQL 3.1.0 are inconsistent with same sql. Spark SQL calculates `{{{}0 in ('00')`{}}} as false, which act different from `{{{}=`{}}} keyword, but Hive calculates true. Hive is compatible with the `{{{}in`{}}} keyword in 3.1.0, but SparkSQL does not.

It's better  when dataTypes of elements in `{{{}In`{}}} expression are the same, it should behaviour as same as BinaryComparison like ` {{{}EqualTo`{}}}.

Test SQL:
{code:java}
scala> spark.sql("select 1 as test where 0 = '00'").show
+----+
|test|
+----+
|   1|
+----+

scala> spark.sql("select 1 as test where 0 in ('00')").show
+----+
|test|
+----+
+----+{code}
 

!image-2023-05-13-13-14-55-853.png!
Bug: Decouple handle command and send response on server side
Status: Open | Priority: Major | Resolution: 
Created: 30/May/23 06:53
Affects Version/s: 3.5.0

SparkConnectStreamHandler treat the request from connect client and send the response back to connect client. SparkConnectStreamHandler hold a component StreamObserver which is used to send response.
So I think we should keep the StreamObserver could be accessed only with SparkConnectStreamHandler.
Bug: reuse SPARK_CONF_DIR config maps between driver and executor
Status: Open | Priority: Major | Resolution: 
Created: 22/May/23 08:38
Affects Version/s: 3.2.4

Currently, Spark on k8s-cluster creates two config maps per application: one for the driver and another for the executor. However the config map for executor is almost identical for config map for driver, there's no need to create there two duplicate config maps. As ConfigMaps are object on K8S, there would be some limitations for ConfigMaps on K8S:
 # more config maps means more objects on etcd, and adds overhead to API server
 # Spark driver pod might be ran under limited permission, which means, it might not be possible to create resources rather than exec pod. Therefore driver might not be allowed to create config maps.

I would submit a pr to reuse SPARK_CONF_DIR config maps for running spark on k8s-cluster mode.
Bug: Support `Column` for SparkConnectColumn.__getitem__
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/23 05:28
Affects Version/s: 3.5.0

Repro:
{code:java}
pser = pd.Series(["a", "b", "c"])
psser = ps.from_pandas(pser)
psser.astype("category")  # internally calls `map_scol[self.spark.column]`{code}
Bug: Support unwrap date type to string type in UnwrapCastInBinaryComparison
Status: Open | Priority: Major | Resolution: 
Created: 25/May/23 20:31
Affects Version/s: 3.5.0

Similar to https://issues.apache.org/jira/browse/SPARK-42597 and others, add support to 
UnwrapCastInBinaryComparison such that it can unwrap date type to string type.
Bug: Support optimize skewed partitions even if introduce extra shuffle
Status: Open | Priority: Major | Resolution: 
Created: 31/May/23 12:59
Affects Version/s: 3.5.0

Similar to [SPARK-33832|https://issues.apache.org/jira/browse/SPARK-33832], OptimizeSkewInRebalancePartitions will not apply if skew mitigation causes a new shuffle.

Test case (data skew in RebalancePartition):
{code:java}
*(2) HashAggregate(keys=[c1#226], functions=[count(1)], output=[c1#226, count(1)#231L])
+- *(2) HashAggregate(keys=[c1#226], functions=[partial_count(1)], output=[c1#226, count#235L])
   +- AQEShuffleRead coalesced
      +- ShuffleQueryStage 0
         +- Exchange hashpartitioning(c1#226, 5), REBALANCE_PARTITIONS_BY_COL, [plan_id=106]
            +- *(1) Project [key#221 AS c1#226]
               +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#221]
                  +- Scan[obj#220] {code}
expect:
{code:java}
HashAggregate(keys=[c1#226], functions=[count(1)], output=[c1#226, count(1)#231L])
+- AQEShuffleRead coalesced
   +- ShuffleQueryStage 1
      +- Exchange hashpartitioning(c1#226, 5), ENSURE_REQUIREMENTS, [plan_id=140]
         +- *(2) HashAggregate(keys=[c1#226], functions=[partial_count(1)], output=[c1#226, count#235L])
            +- AQEShuffleRead coalesced and skewed
               +- ShuffleQueryStage 0
                  +- Exchange hashpartitioning(c1#226, 5), REBALANCE_PARTITIONS_BY_COL, [plan_id=106]
                     +- *(1) Project [key#221 AS c1#226]
                        +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#221]
                           +- Scan[obj#220] {code}
Bug: Support `pyspark.ml.feature.Bucketizer` and `pyspark.mllib.stat.KernelDensity` to work with Spark Connect.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/23 11:02
Affects Version/s: 4.0.0

Repro: run `DataFramePlotParityTests.test_compute_hist_multi_columns` or `

SeriesPlotMatplotlibParityTests.test_kde_plot`
Bug: Fix behavior difference for compare binary functions.
Status: Open | Priority: Major | Resolution: 
Created: 30/May/23 05:49
Affects Version/s: 3.5.0

In [https://github.com/apache/spark/pull/41362,] we add `result = result.fillna(False)` for filling the gap between pandas <> pandas API on Spark, but it should be internally fixed from Spark Connect side. Please refer to the reproducible code below:

 
{code:java}
import pandas as pd
import pyspark.pandas as ps
from pyspark.sql.utils import pyspark_column_op

pser = pd.Series([None, None, None])
psser = ps.from_pandas(pser)
pyspark_column_op("__ge__")(psser, psser)
# Wrong result:
#  0    None
#  1    None
#  2    None
#  dtype: object

# Expected result:
pser > pser
#  0    False
#  1    False
#  2    False
dtype: bool{code}
Bug: Match `GroupBy.nth` behavior with new pandas behavior
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/23 05:09
Affects Version/s: 4.0.0

Match behavior with https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#dataframegroupby-nth-and-seriesgroupby-nth-now-behave-as-filtrations
Bug: Enable GroupByTests for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 02:35
Affects Version/s: 4.0.0

test list:
 * test_prod
 * test_nth
 * test_mad
 * test_basic_stat_funcs
 * test_groupby_multiindex_columns
 * test_apply_without_shortcut
 * test_mean
 * test_apply
Bug: Memory leak in Spark UI
Status: Open | Priority: Major | Resolution: 
Created: 16/May/23 09:35
Affects Version/s: 2.4.4

We have a distributed Spark application running on Azure HDInsight using Spark version 2.4.4.

After a few days of active processing on our application, we have noticed that the GC CPU time ratio of the driver is close to 100%. We suspected a memory leak. Thus, we have produced a heap dump and analyzed it using Eclipse Memory Analyzer.

Here is some interesting data from the driver's heap dump (heap size is 8 GB):
 * The estimated retained heap size of String objects (~5M instances) is 3.3 GB. It seems that most of these instances correspond to spark events.
 * Spark UI's AppStatusListener instance estimated retained size is 1.1 GB.
 * The number of LiveJob objects with status "RUNNING" is 18K, knowing that there shouldn't be more than 16 live running jobs since we use a fixed size thread pool of 16 threads to run spark queries.
 * The number of LiveTask objects is 485K.
 * The AsyncEventQueue instance associated to the AppStatusListener has a value of 854 for dropped events count and a value of 10001 for total events count, knowing that the dropped events counter is reset every minute and that the queue's default capacity is 10000.

We think that there is a memory leak in Spark UI. Here is our analysis of the root cause of this leak:
 * AppStatusListener is notified of Spark events using a bounded queue in AsyncEventQueue.
 * AppStatusListener updates its state (kvstore, liveTasks, liveStages, liveJobs, ...) based on the received events. For example, onTaskStart adds a task to liveTasks map and onTaskEnd removes the task from liveTasks map.
 * When the rate of events is very high, the bounded queue in AsyncEventQueue is full, some events are dropped and don't make it to AppStatusListener.
 * Dropped events that signal the end of a processing unit prevent the state of AppStatusListener from being cleaned. For example, a dropped onTaskEnd event, will prevent the task from being removed from liveTasks map, and the task will remain in the heap until the driver's JVM is stopped.

We were able to confirm our analysis by reducing the capacity of the AsyncEventQueue (spark.scheduler.listenerbus.eventqueue.capacity=10). After having launched many spark queries using this config, we observed that the number of active jobs in Spark UI increased rapidly and remained high even though all submitted queries have completed. We have also noticed that some executor task counters in Spark UI were negative, which confirms that AppStatusListener state does not accurately reflect the reality and that it can be a victim of event drops.

Suggested fix:
There are some limits today on the number of "dead" objects in AppStatusListener's maps (for example: spark.ui.retainedJobs). We suggest enforcing another configurable limit on the number of total objects in AppStatusListener's maps and kvstore. This should limit the leak in the case of high events rate, but AppStatusListener stats will remain inaccurate.
Bug: Enable DataFramePlotMatplotlibTests for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 02:27
Affects Version/s: 4.0.0

test list:
 * test_area_plot
 * test_area_plot_stacked_false
 * test_area_plot_y
 * test_bar_plot
 * test_bar_with_x_y
 * test_barh_plot_with_x_y
 * test_barh_plot
 * test_line_plot
 * test_pie_plot
 * test_scatter_plot
 * test_hist_plot
 * test_kde_plot
Bug: Fix mismatched column list error on INSERT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/23 17:32
Affects Version/s: 3.4.0

This error message is pretty bad, and common
"_LEGACY_ERROR_TEMP_1038" : {
"message" : [
"Cannot write to table due to mismatched user specified column size(<columnSize>) and data column size(<outputSize>)."
]
},

It can perhaps be merged with this one - after giving it an ERROR_CLASS

"_LEGACY_ERROR_TEMP_1168" : {
"message" : [
"<tableName> requires that the data to be inserted have the same number of columns as the target table: target table has <targetColumns> column(s) but the inserted data has <insertedColumns> column(s), including <staticPartCols> partition column(s) having constant value(s)."
]
},



Repro:

CREATE TABLE tabtest(c1 INT, c2 INT);


INSERT INTO tabtest SELECT 1;

`spark_catalog`.`default`.`tabtest` requires that the data to be inserted have the same number of columns as the target table: target table has 2 column(s) but the inserted data has 1 column(s), including 0 partition column(s) having constant value(s).

INSERT INTO tabtest(c1) SELECT 1, 2, 3;
Cannot write to table due to mismatched user specified column size(1) and data column size(3).; line 1 pos 24
Bug: Make DisableUnnecessaryBucketedScan smart with table cache
Status: Open | Priority: Major | Resolution: 
Created: 09/May/23 06:22
Affects Version/s: 3.5.0

If a bucket scan has no interesting partition or contains shuffle exchange, then we would disable it. But If the bucket scan is inside table cache, the cached plan would be accessed multi-times, then we should not disable it as it could preserve output partitioning and more likely be reused.
Bug: Enable OpsOnDiffFramesGroupByTests.test_groupby_different_lengths for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 11/May/23 12:17
Affects Version/s: 4.0.0

Enable OpsOnDiffFramesGroupByTests.test_groupby_different_lengths for pandas 2.0.0.
Bug: Enable CategoricalTests for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 02:38
Affects Version/s: 4.0.0

test list:
 * test_factorize
 * test_as_ordered_unordered
 * test_categories_setter
 * test_remove_categories
 * test_groupby_apply_without_shortcut
Bug: Enable SeriesDateTimeTests for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 02:21
Affects Version/s: 4.0.0

test list:
 * test_day
 * test_dayofweek
 * test_dayofyear
 * test_days_in_month
 * test_daysinmonth
 * test_hour
 * test_microsecond
 * test_minute
 * test_month
 * test_quarter
 * test_second
 * test_wrrkday
 * test_year
Bug: Support correlated columns in join ON conditions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/23 20:03
Affects Version/s: 3.4.0

Subqueries that have joins with outer references in join conditions should be supported.

For example:

 
{code:java}
spark-sql (default)> create view t0(t0a) as values (0), (1), (2);
spark-sql (default)> create view t1(t1a) as values (1), (2), (3);
spark-sql (default)> create view t2(t2a) as values (2), (3), (4);
spark-sql (default)> select * from t0 join lateral (select * from t1 join t2 on t1a = t2a and t1a = t0a);
[UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.CORRELATED_REFERENCE] Unsupported subquery expression: Expressions referencing the outer query are not supported outside of WHERE/HAVING clauses: "((t1a = t2a) AND (t1a = t0a))".; line 1 pos 48;
Project [t0a#15, t1a#17, t2a#19]
+- LateralJoin lateral-subquery#14 [t0a#15], Inner
   :  +- SubqueryAlias __auto_generated_subquery_name
   :     +- Project [t1a#17, t2a#19]
   :        +- Join Inner, ((t1a#17 = t2a#19) AND (t1a#17 = outer(t0a#15)))
   :           :- SubqueryAlias spark_catalog.default.t1
   :           :  +- View (`spark_catalog`.`default`.`t1`, [t1a#17])
   :           :     +- Project [cast(col1#18 as int) AS t1a#17]
   :           :        +- LocalRelation [col1#18]
   :           +- SubqueryAlias spark_catalog.default.t2
   :              +- View (`spark_catalog`.`default`.`t2`, [t2a#19])
   :                 +- Project [cast(col1#20 as int) AS t2a#19]
   :                    +- LocalRelation [col1#20]
   +- SubqueryAlias spark_catalog.default.t0
      +- View (`spark_catalog`.`default`.`t0`, [t0a#15])
         +- Project [cast(col1#16 as int) AS t0a#15]
            +- LocalRelation [col1#16] {code}
Here, the subquery has a join with join condition referencing the outer table t0. This should be handled very similarly to how we handle correlation in Filter predicates.
Bug: Spark Connect - prefer to pass around SessionHolder / ExecuteHolder more
Status: Open | Priority: Major | Resolution: 
Created: 23/May/23 16:16
Affects Version/s: 3.5.0

Right now, we pass around inidvidual things like sessionId, userId etc. in multiple places. This needs to the need of additional threading of parameters quite often when something new is added.

Better pass around SessionHolder and ExecutePlanHolder, where accessor and utility functions can be added.
Bug: Enable SeriesTests for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 02:07
Affects Version/s: 4.0.0

test list:
 * test_value_counts
 * test_append
 * test_astype
 * test_between
 * test_mad
 * test_quantile
 * test_rank
 * test_between_time
 * test_iteritems
 * test_product
 * test_factorize
Bug: Enable DataFrameSlowTests for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 02:31
Affects Version/s: 4.0.0

test list:
 * test_describe
 * test_between_time
 * test_product
 * test_iteritems
 * test_mad
 * test_cov
 * test_quantile
Bug: Spark Connect - decouple query execution from RPC handler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/May/23 16:10
Affects Version/s: 3.5.0

Move actual query execution out of the RPC handler callback. This allows:
 * (immediately) better control over query cancellation, by interrupting the execution thread.
 * design changes to the RPC interface to allow different execution models than stream-push from server.
Bug: withColumnRenamed duplicates columns if new column already exists
Status: Open | Priority: Major | Resolution: 
Created: 15/May/23 20:58
Affects Version/s: 3.4.0

withColumnRenamed should either replace the column when new column already exists or should specify the specificity in the documentation. See the code below as an example of the current state.

{code:python}
from pyspark.sql import SparkSession

spark = SparkSession.builder.master("local[1]").appName("local-spark-session").getOrCreate()

df = spark.createDataFrame([(1, 0.5, 0.4), (2, 0.5, 0.8)], ["id", "score", "test_score"])
r = df.withColumnRenamed("test_score", "score")
print(r)  # DataFrame[id: bigint, score: double, score: double]

# pyspark.sql.utils.AnalysisException: Reference 'score' is ambiguous, could be: score, score.
print(r.select("score"))
{code}
Bug: Add optional pattern for Catalog.listDatabases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 10:11
Affects Version/s: 3.5.0

Currently, the syntax

{code:java}
SHOW DATABASES LIKE pattern
{code}

supports a optional pattern.
But the Catalog.listDatabases missing the function.
In fact, the optional pattern is very useful.
Bug: Add optional pattern for Catalog.listCatalogs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/23 08:31
Affects Version/s: 3.5.0

Currently, the syntax 
{code:java}
SHOW CATALOGS LIKE pattern
{code}
 supports a optional pattern.
But the catalog.listCatalogs missing the function.
In fact, the optional pattern is very useful.
Bug: Add a max distance argument to the levenshtein() function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/23 16:09
Affects Version/s: 3.4.0

Currently, Spark's levenshtein(str1, str2) function can be very inefficient for long strings. Many other databases which support this type of built-in function also take a third argument which signifies a maximum distance after which it is okay to terminate the algorithm.

For example something like

{code:sql}
levenshtein(str1, str2[, max_distance])
{code}

the function stops computing the distant once the max values is reached.
See postgresql for an example of a 3 argument [levenshtein|https://www.postgresql.org/docs/current/fuzzystrmatch.html#id-1.11.7.26.7].
Bug: Enable DataFramePlotMatplotlibTests.test_barh_plot_with_x_y for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/May/23 06:42
Affects Version/s: 3.5.0

Enable DataFramePlotMatplotlibTests.test_barh_plot_with_x_y for pandas 2.0.0.
Bug: Enable DataFramePlotMatplotlibTests.test_bar_with_x_y for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/May/23 06:40
Affects Version/s: 3.5.0

Enable DataFramePlotMatplotlibTests.test_bar_with_x_y for pandas 2.0.0.