av9ash/GitBugs huggingface dataset:
Bug: Mutilple tables join with limit when AE is enabled and one table is skewed
Status: Open | Priority: Critical | Resolution: 
Created: 19/Apr/23 08:48
Affects Version/s: 3.4.0

When we test AE in Spark3.4.0 with the following case, we find If we disable AE or enable Ae but disable skewJoin, the sql will finish in 20s, but if we enable AE and enable skewJoin，it will take very long time.

The test case:
{code:java}
###uncompress the part-m-***.zip attachment, and put these files under '/tmp/spark-warehouse/data/' dir.

create table source_aqe(c1 int,c18 string) using csv options(path 'file:///tmp/spark-warehouse/data/');
create table hive_snappy_aqe_table1(c1 int)stored as PARQUET partitioned by(c18 string); 
insert into table hive_snappy_aqe_table1 partition(c18=1)select c1 from source_aqe;
insert into table hive_snappy_aqe_table1 partition(c18=2)select c1 from source_aqe limit 120000;
insert into table hive_snappy_aqe_table1 partition(c18=3)select c1 from source_aqe limit 150000;create table hive_snappy_aqe_table2(c1 int)stored as PARQUET partitioned by(c18 string); 
insert into table hive_snappy_aqe_table2 partition(c18=1)select c1 from source_aqe limit 160000;
insert into table hive_snappy_aqe_table2 partition(c18=2)select c1 from source_aqe limit 120000;create table hive_snappy_aqe_table3(c1 int)stored as PARQUET partitioned by(c18 string); 
insert into table hive_snappy_aqe_table3 partition(c18=1)select c1 from source_aqe limit 160000;
insert into table hive_snappy_aqe_table3 partition(c18=2)select c1 from source_aqe limit 120000;
set spark.sql.adaptive.enabled=false;
set spark.sql.adaptive.forceOptimizeSkewedJoin = false;
set spark.sql.adaptive.skewJoin.skewedPartitionFactor=1;
set spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=10KB;
set spark.sql.adaptive.advisoryPartitionSizeInBytes=100KB;
set spark.sql.autoBroadcastJoinThreshold = 51200;
 
###it will finish in 20s 
select * from hive_snappy_aqe_table1 join hive_snappy_aqe_table2 on hive_snappy_aqe_table1.c18=hive_snappy_aqe_table2.c18 join hive_snappy_aqe_table3 on hive_snappy_aqe_table1.c18=hive_snappy_aqe_table3.c18 limit 10;


set spark.sql.adaptive.enabled=true;
set spark.sql.adaptive.forceOptimizeSkewedJoin = true;
set spark.sql.adaptive.skewJoin.skewedPartitionFactor=1;
set spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=10KB;
set spark.sql.adaptive.advisoryPartitionSizeInBytes=100KB;
set spark.sql.autoBroadcastJoinThreshold = 51200;
###it will take very long time 
select * from hive_snappy_aqe_table1 join hive_snappy_aqe_table2 on hive_snappy_aqe_table1.c18=hive_snappy_aqe_table2.c18 join hive_snappy_aqe_table3 on hive_snappy_aqe_table1.c18=hive_snappy_aqe_table3.c18 limit 10;
{code}
Bug: Connect Server Can't Handle Exception With Null Message Normally
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 13/Apr/23 13:04
Affects Version/s: 3.4.0

!image-2023-04-13-21-05-43-617.png|width=626,height=310!!image-2023-04-13-21-05-16-994.png|width=621,height=294!

When Server Throw Exception which without message Like NPE. The setMessage method will report NPE again. But can't throw to client.
Bug: Connect to multiple hive metastore using single sparkcontext (i.e without stopping it")
Status: Resolved | Priority: Critical | Resolution: Invalid
Created: 03/Apr/23 23:19
Affects Version/s: 3.2.0

I wanted to join 2 hive tables which are stored in 2 different hive metastores.
I would like fetch the first table, create a tempview. Change hive metastore uri, connect to that metastore and fetch the second table , create a tempview and then use tempviews to do further. 

I launched 2 different instances of hive on my local , individually i'm able to connect to both, but when i try to connect to them one after other, the second time the old URI's is getting used.

Any workaround this.

Please help out :\
Bug: Executor obtained error information 
Status: Open | Priority: Major | Resolution: 
Created: 20/Apr/23 15:56
Affects Version/s: 3.1.1

Spark on Yarn Cluster

When multiple executors exist on a node, and the same block exists on both executors, with some in memory and some on disk.

Probabilistically, the executor failed to obtain the block,throw Exception:

java.lang.ArrayIndexOutofBoundsException: 0

    at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBlocks$1(TorrentBroadcast.scala:183)

 

Next, I will replay the process of the problem occurring: 

step 1:

The executor requests the driver to obtain block information(locationsAndStatusOption). The input parameters are BlockId and the host of its own node. Please note that it does not carry port information

line:1092

!image-2023-04-21-00-24-22-059.png!

step 2:

On the driver side, the driver obtains all blockManagers holding the block based on the BlockId. For non remote shuffle scenarios, the driver will retrieve the first one with the blockId and blockManager from the locations

Assuming that there are two BlockManagers holding the BlockId on this node, BM-1 holds the Block and stores it in memory, and BM-2 holds the Block and stores it in disk

Assuming the returned status is of type memory and its disksize is 0

line: 852, 856

!image-2023-04-21-00-30-41-851.png!

step 3:

This method will return a BlockLocationsAndStatus object. If there are BMs using disk, the disk's path information will be stored in localDirs

!image-2023-04-21-00-50-10-918.png!

step 4:

When the executor obtains locationsAndStatusOption, localDirs is not empty, but status.diskSize is 0

line: 1102

!image-2023-04-21-00-54-11-968.png!

step 5:

The readDiskBlockFromSameHostExecutor only determines whether the Block file exists, and then directly uses the incoming blocksize to read the byte array. If the blocksize is 0, it returns an empty byte array

Only checked if the file exists

line: 1234, 1240

!image-2023-04-21-00-57-29-140.png!

Taking values from an empty array, causing an out of bounds problem
Bug: Add labels to identify UDF 
Status: Open | Priority: Major | Resolution: 
Created: 13/Apr/23 19:43
Affects Version/s: 4.0.0

I wish there is a convenient way to report the UDFs executed by spark application. This is a useful information to analyze the performance and possible optimizations.

Given an application eventlog, I'm not seeing anything in the plans that can easily identify UDFs.


I see that SPARK-34388  added "source" to the expressionInfo, but this extra-info is not propagated to the "explain" or the eventlog.
I suggest that UDF shows in the eventlog as something like 


{code:scala}
explain sql("SELECT myUdf1(myUdf2(1))"))
myUdf1#scalaUDF(myUdf2#scalaUDF(1))
{code}
Bug: JVM Client throw StreamingQueryException when error handling is implemented
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/23 20:49
Affects Version/s: 3.5.0

Currently the awaitTermination() method of connect's JVM client's StreamingQuery won't throw error when there is an exception. 

 

In Python connect this is directly handled by python client's error-handling framework but such is not existed in JVM client right now.

 

We should verify it works when JVM adds that
Bug: Fix DROP table behavior in session catalog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/23 19:16
Affects Version/s: 3.4.0

DROP table behavior is not working correctly in 3.4.0 because we always invoke V1 drop logic if the identifier looks like a V1 identifier. This is a big blocker for external data sources that provide custom session catalogs.

See [here|https://github.com/apache/spark/pull/37879/files#r1170501180] for details.
Bug: Eliminate Union if filters have the same child plan
Status: Open | Priority: Major | Resolution: 
Created: 04/Apr/23 05:48
Affects Version/s: 3.5.0

There are a lot of SQL in user scenarios. Take an example,
SELECT
  ss_item_sk,
  ss_ticket_number,
  ss_customer_sk
FROM store_sales
  LEFT OUTER JOIN store_returns
    ON (sr_item_sk = ss_item_sk AND sr_ticket_number = ss_ticket_number)
WHERE sr_return_amt > 10000
UNION ALL
SELECT
  ss_item_sk,
  ss_ticket_number,
  ss_customer_sk
FROM store_sales
  LEFT OUTER JOIN store_returns
    ON (sr_item_sk = ss_item_sk AND sr_ticket_number = ss_ticket_number)
WHERE sr_return_amt < 1000

We can simplify this SQL as
SELECT
  ss_item_sk,
  ss_ticket_number,
  ss_customer_sk
FROM store_sales
  LEFT OUTER JOIN store_returns
    ON (sr_item_sk = ss_item_sk AND sr_ticket_number = ss_ticket_number)
WHERE sr_return_amt > 10000 or sr_return_amt < 1000
Bug: Improve the performance of MapOutputTracker.updateMapOutput
Status: Resolved | Priority: Major | Resolution: Done
Created: 05/Apr/23 23:09
Affects Version/s: 3.3.2

Inside of MapOutputTracker, there is a line of code which does a linear find through a mapStatuses collection: https://github.com/apache/spark/blob/cb48c0e48eeff2b7b51176d0241491300e5aad6f/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L167  (plus a similar search a few lines down at https://github.com/apache/spark/blob/cb48c0e48eeff2b7b51176d0241491300e5aad6f/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L174)

This scan is necessary because we only know the mapId of the updated status and not its mapPartitionId.

We perform this scan once per migrated block, so if a large proportion of all blocks in the map are migrated then we get O(n^2) total runtime across all of the calls.

I think we might be able to fix this by extending ShuffleStatus to have an OpenHashMap mapping from mapId to mapPartitionId.
Bug: When CTAS with USING fails to store metadata in metastore, data gets left around
Status: Open | Priority: Major | Resolution: 
Created: 15/Apr/23 01:31
Affects Version/s: 3.5.0

For example:
{noformat}
drop table if exists parquet_ds1;

-- try creating table with invalid column name
-- use 'using parquet' to designate the data source
create table parquet_ds1 using parquet as
select id, date'2018-01-01' + make_dt_interval(0, id)
from range(0, 10);

Cannot create a table having a column whose name contains commas in Hive metastore. Table: `spark_catalog`.`default`.`parquet_ds1`; Column: DATE '2018-01-01' + make_dt_interval(0, id, 0, 0.000000)

-- show that table did not get created
show tables;


-- try again with valid column name
-- spark will complain that directory already exists
create table parquet_ds1 using parquet as
select id, date'2018-01-01' + make_dt_interval(0, id) as ts
from range(0, 10);

[LOCATION_ALREADY_EXISTS] Cannot name the managed table as `spark_catalog`.`default`.`parquet_ds1`, as its associated location 'file:/Users/bruce/github/spark_upstream/spark-warehouse/parquet_ds1' already exists. Please pick a different table name, or remove the existing location first.
org.apache.spark.SparkRuntimeException: [LOCATION_ALREADY_EXISTS] Cannot name the managed table as `spark_catalog`.`default`.`parquet_ds1`, as its associated location 'file:/Users/bruce/github/spark_upstream/spark-warehouse/parquet_ds1' already exists. Please pick a different table name, or remove the existing location first.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.locationAlreadyExists(QueryExecutionErrors.scala:2804)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.validateTableLocation(SessionCatalog.scala:414)
	at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:176)
...
{noformat}
One must manually remove the directory {{spark-warehouse/parquet_ds1}} before the {{create table}} command will succeed.

It seems that datasource table creation runs the data-creation job first, then stores the metadata into the metastore.

When using Spark to create Hive tables, the issue does not happen:
{noformat}
drop table if exists parquet_hive1;

-- try creating table with invalid column name,
-- but use 'stored as parquet' instead of 'using'
create table parquet_hive1 stored as parquet as
select id, date'2018-01-01' + make_dt_interval(0, id)
from range(0, 10);

Cannot create a table having a column whose name contains commas in Hive metastore. Table: `spark_catalog`.`default`.`parquet_hive1`; Column: DATE '2018-01-01' + make_dt_interval(0, id, 0, 0.000000)

-- try again with valid column name. This will succeed;
create table parquet_hive1 stored as parquet as
select id, date'2018-01-01' + make_dt_interval(0, id) as ts
from range(0, 10);
{noformat}

It seems that Hive table creation stores metadata into the metastore first, then runs the data-creation job.
Bug: User-defined output metadata path (_spark_metadata)
Status: Open | Priority: Major | Resolution: 
Created: 15/Apr/23 17:29
Affects Version/s: 3.4.0

Currently path of metadata of output checkpoint is hardcoded. The metadata is saved in output path in _spark_metadata folder. It's a constraint on structure of paths, that might be easily relaxed by parametrisable path of output metadata. It would help with issues like [changing output directory of spark streaming job|https://kb.databricks.com/en_US/streaming/file-sink-streaming], [two jobs writing to the same output path|https://issues.apache.org/jira/browse/SPARK-30542] or [partition discovery|https://stackoverflow.com/questions/61904732/is-it-possible-to-change-location-of-spark-metadata-folder-in-spark-structured/61905158]. It would also help with separation of metadata from data in path structure.

The main target of change is getMetadataLogPath method in FileStreamSink. It has got access to sqlConf, so this method can override the default _spark_metadata path if defined it config. Introduction of parametrised metadata path needs reconsidering of meaning of  hasMetadata method in FileStreamSink.
Bug: Improve ObjectHashAggregateExec performance for high cardinality
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/23 09:53
Affects Version/s: 3.5.0

The `ObjectHashAggregateExec` has three preformance issues:
 - heavy overhead of scala sugar in `createNewAggregationBuffer`
 - unnecessary grouping key comparation after fallback to sort based aggregator
 - the aggregation buffer in sort based aggregator is not reused for all rest rows
Bug: Data lost from the table if the INSERT OVERWRITE query fails
Status: Open | Priority: Major | Resolution: 
Created: 12/Apr/23 10:03
Affects Version/s: 3.4.0

When we run an INSERT OVERWRITE query for an unpartitioned table on Spark-3, Spark has the following behavior:

1) It will first clean up all the data from the actual table path.

2) It will then launch a job that performs the actual insert.

 

There are 2 major issues with this approach:

1) If the insert job launched in step 2 above fails for any reason, the data from the original table is lost. 

2) If the insert job in step 2 above takes a huge time to complete, then table data is unavailable to other readers for the entire duration the job takes.

This behavior is the same even for the partitioned tables when using static partitioning. For dynamic partitioning, we do not delete the table data before the job launch.

 

Is there a reason as to why we perform this delete before the job launch and not as part of the Job commit operation? This issue is not there with Hive - where the data is cleaned up as part of the Job commit operation probably. As part of SPARK-19183, we did add a new hook in the commit protocol for this exact same purpose, but seems like its default behavior is still to delete the table data before the job launch.
Bug: df.describe() method may- return wrong result if the last RDD is RDD[UnsafeRow]
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/23 02:50
Affects Version/s: 3.3.2

When calling the df.describe() method, the result  maybe wrong when the last RDD is RDD[UnsafeRow]. It is because the UnsafeRow will be released after the row is used.
Bug: Should not handle the COUNT bug when the GROUP BY clause of a correlated scalar subquery is non-empty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/23 19:38
Affects Version/s: 3.2.0

From [~allisonwang-db] :

There is no COUNT bug when the correlated equality predicates are also in the group by clause. However, the current logic to handle the COUNT bug still adds default aggregate function value and returns incorrect results.

 
{code:java}
create view t1(c1, c2) as values (0, 1), (1, 2);
create view t2(c1, c2) as values (0, 2), (0, 3);

select c1, c2, (select count(*) from t2 where t1.c1 = t2.c1 group by c1) from t1;

-- Correct answer: [(0, 1, 2), (1, 2, null)]
+---+---+------------------+
|c1 |c2 |scalarsubquery(c1)|
+---+---+------------------+
|0  |1  |2                 |
|1  |2  |0                 |
+---+---+------------------+
 {code}
 

This bug affects scalar subqueries in RewriteCorrelatedScalarSubquery, but lateral subqueries handle it correctly in DecorrelateInnerQuery. Related: https://issues.apache.org/jira/browse/SPARK-36113
Bug: Mismatch of field name in log event writer and parser for push shuffle metrics
Status: Open | Priority: Major | Resolution: 
Created: 11/Apr/23 23:08
Affects Version/s: 3.4.0

For push based shuffle metrics, when writting out the event to log file, the field name is "Push Based Shuffle", in [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala#L548]

But when parsing it out in SHS, the expected field name is "Shuffle Push Read Metrics", as shown [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala#L1264]

This mismatch makes all the push shuffle metrics 0 from SHS rest calls.
Bug: Refine `column_op` to use lambda function instead of Column API.
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 17/Apr/23 06:21
Affects Version/s: 3.5.0

Refining `column_op(Column.__eq__)(left, right)` to use lambda function such as `column_op(lambda x, y: x.__eq__(y))(left, right)`
Bug: Generate proper warning on different behavior with numeric_only
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 26/Apr/23 05:12
Affects Version/s: 4.0.0

Should enable test below:
{code:java}
pdf = pd.DataFrame([("1", "2"), ("0", "3"), ("2", "0"), ("1", "1")], columns=["a", "b"])
psdf = ps.from_pandas(pdf)
self.assert_eq(pdf.cov(), psdf.cov()) {code}
Bug: Investigate DataFrame.sort_values with pandas behavior.
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 25/Apr/23 13:07
Affects Version/s: 4.0.0

{code:java}
import pandas as pd
pdf = pd.DataFrame(
    {
        "a": pd.Categorical([1, 2, 3, 1, 2, 3]),
        "b": pd.Categorical(
            ["b", "a", "c", "c", "b", "a"], categories=["c", "b", "d", "a"]
        ),
    },
)
pdf.groupby("a").apply(lambda x: x).sort_values(["a"])

Traceback (most recent call last):
...
ValueError: 'a' is both an index level and a column label, which is ambiguous. {code}
We should investigate this issue whether this is intended behavior or just bug in pandas.
Bug: DataSourceV2 is hard to be implemented without following V1
Status: Open | Priority: Major | Resolution: 
Created: 16/Apr/23 13:53
Affects Version/s: 3.4.0

h1. Description

The current interface of DataSourceV2 becomes overly complicated than the Spark 2.x versions. To implement under the DataSourceV2, user needs to learn not only the V2 APIs and interfaces. But also the DataSourceV1 (as it is a failback version).
h2. Interface Gaps

There is no easy way and clear examples on how to implement both for a new dataSource. For example, the examples in standard spark repo like orc, parquet, json has a FileFormat interface for V1 while all these are not feasible to be followed since the SPI is hard-code as `DefaultSource` instead of dynamic loading if from user provided class outside the Spark Repo. Different data sources are not strictly following a same pattern in V1 and not decoupled well with customized logic within it.

 
h2. Loss of simple layer over different kinds of dataSource

With original V1, user can actually implement a new wrapper on top of orc/parquet easily with Relation Interface. The DataSourceV2 again here becomes too low level and hard to be used in this case.

 
h2. No explicit guidance

The functionality interfaces are not well organized which forces the reader spend lots of time to understand the commit history, existing patterns as well.
Bug: spark reader csv and json support wholetext parameters
Status: Open | Priority: Major | Resolution: 
Created: 28/Apr/23 12:23
Affects Version/s: 3.5.0

FTPInputStream used by Hadoop FTPFileSystem does not support seek, and spark HadoopFileLinesReader fails to be read. 

Support to read the entire file, and then split lines, avoid reading failure

 

[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPInputStream.java]

 

[~cloud_fan]
Bug: Refactoring similar error classes such as `NOT_XXX`.
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 04/Apr/23 01:17
Affects Version/s: 3.5.0

We should consolidate the error classes that have a similar error messages into single error class, or classify them into a main-sub error class structure.

*NOTE:* This refactoring should be started after all other initial migration is done.
Bug: Make DataFrameGroupBy.sum support for string type columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/23 16:06
Affects Version/s: 4.0.0

From pandas 2.0.0, DataFrameGroupBy.sum also works for string type columns:
{code:java}
>>> psdf
   A    B  C      D
0  1  3.1  a   True
1  2  4.1  b  False
2  1  4.1  b  False
3  2  3.1  a   True
>>> psdf.groupby("A").sum().sort_index()
     B  D
A
1  7.2  1
2  7.2  1
>>> psdf.to_pandas().groupby("A").sum().sort_index()
     B   C  D
A
1  7.2  ab  1
2  7.2  ba  1 {code}
Bug: PySpark 3.4.0 cannot convert timestamp-typed objects to pandas with pandas 2.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/23 13:23
Affects Version/s: 4.0.0

{code}
In [1]: from pyspark.sql import SparkSession

In [2]: session = SparkSession.builder.appName("test").getOrCreate()
23/04/19 09:21:42 WARN Utils: Your hostname, albatross resolves to a loopback address: 127.0.0.2; using 192.168.1.170 instead (on interface enp5s0)
23/04/19 09:21:42 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/04/19 09:21:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

In [3]: session.sql("select now()").toPandas()
{code}

Results in:

{code}
...
TypeError: Casting to unit-less dtype 'datetime64' is not supported. Pass e.g. 'datetime64[ns]' instead.
{code}