av9ash/GitBugs huggingface dataset:
Bug: The saveAsTable method fails to be executed, resulting in data file loss
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/22 06:02
Affects Version/s: 3.3.1

We have a problem in the production environment. 
The code is: df.write.mode(SaveMode.Overwrite).saveAsTable("xxx").
When the saveAsTable method is executing, an executor exits due to OOM, causing half of the data file to be written on the hdfs, but subsequent spark retries fail again. 

It is very similar to the scenario described in SPARK-22504, but it really happened.

I read the source code. Why does Spark need to delete the table first and then execute the plan? What if the execution fails after deleting the table?

I know the attitude of the community, but this method of deleting tables first is too risky. Can we adopt the following processing methods like Hive,
1. WRITE: create and write data to tempTable
2. SWAP: swap temptable1 with targetTable by using rename operation
3. CLEAN: clean up old data
Bug: DataFrame creation from 2d NumPy array with arbitrary columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Nov/22 21:26
Affects Version/s: 3.4.0

Currently, DataFrame creation from 2d ndarray works only with 2 columns. We should provide complete support for DataFrame creation with 2d ndarray.

For example, the test case below should work as shown below.

 
{code:java}
>>> spark.createDataFrame(np.arange(100).reshape([10,10])).show()
+---+---+---+---+---+---+---+---+---+---+                                       
| _1| _2| _3| _4| _5| _6| _7| _8| _9|_10|
+---+---+---+---+---+---+---+---+---+---+
|  0|  1|  2|  3|  4|  5|  6|  7|  8|  9|
| 10| 11| 12| 13| 14| 15| 16| 17| 18| 19|
| 20| 21| 22| 23| 24| 25| 26| 27| 28| 29|
| 30| 31| 32| 33| 34| 35| 36| 37| 38| 39|
| 40| 41| 42| 43| 44| 45| 46| 47| 48| 49|
| 50| 51| 52| 53| 54| 55| 56| 57| 58| 59|
| 60| 61| 62| 63| 64| 65| 66| 67| 68| 69|
| 70| 71| 72| 73| 74| 75| 76| 77| 78| 79|
| 80| 81| 82| 83| 84| 85| 86| 87| 88| 89|
| 90| 91| 92| 93| 94| 95| 96| 97| 98| 99|
+---+---+---+---+---+---+---+---+---+---+
  {code}
Bug: OOM when filter pushdown `last_day` function
Status: Open | Priority: Major | Resolution: 
Created: 28/Nov/22 12:42
Affects Version/s: 3.3.1

Using the following transformation on Spark 3.3.1:
{code:java}
df.where($"date" === last_day($"date")) {code}
Where `df` is a dataframe created from a set Parquet files.  I'm trying to filter dates where they match with the last day of the month of where `date` happened.

Executors are dying with the following error:
{code:java}
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.regex.Pattern.compile(Pattern.java:1722) ~[?:1.8.0_252]
at java.util.regex.Pattern.<init>(Pattern.java:1352) ~[?:1.8.0_252]
at java.util.regex.Pattern.compile(Pattern.java:1028) ~[?:1.8.0_252] {code}
By *disabling* the predicate pushdown rule, the job works normally.

 Also, this works normally on Spark 3.3.0. I also couldn't verify other date functions failing on the same way.
Bug: applyInPandasWithState can produce incorrect key value in user function for timed out state
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/22 07:01
Affects Version/s: 3.4.0

We observed the issue that user function retrieves incorrect key in user function for timed out state. After RCA we figured out this could happen when the columns of grouping keys are not placed sequentially at earliest place.
Bug: When using `db_ name.temp_ table_name`, an exception will be thrown
Status: Open | Priority: Major | Resolution: 
Created: 23/Nov/22 06:49
Affects Version/s: 3.2.0

SQL1:
{code:java}
with table_hive1 as(select * from db1.table_hive)
select * from db1.table_hive1; {code}
 

It will throw exception `org.apache.spark.sql.AnalysisException: Table or view not found: db1.table_hive1;`but spark in 2.4.3 work well.

SQL2:
{code:java}
with table_hive1 as(select * from db1.table_hive)
select * from table_hive1; {code}
It work well.
I'm a little confused. Is this syntax with database name not supported.

you can run like this:
{code:java}
create db1.table_hive(age int, name string);
insert into db1.table_hive values(1, 'a');
insert into db1.table_hive values(2, 'b');
insert into db1.table_hive values(3, 'c');
with table_hive1 as(select * from db1.table_hive)
select * from db1.table_hive1; {code}
Bug: Performance issue when Spark SQL connects with TeraData 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Nov/22 11:30
Affects Version/s: 2.4.4

We are connecting Tera data from spark SQL with below API

{color:#ff8b00}Dataset<Row> jdbcDF = spark.read().jdbc(connectionUrl, tableQuery, connectionProperties);{color}

We are facing one issue when we execute above logic on large table with million rows every time we are seeing below extra query is executing every time as this resulting performance hit on DB.

This below information we got from DBA. We dont have any logs on SPARK SQL.

SELECT 1 FROM ONE_MILLION_ROWS_TABLE;
|1|
|1|
|1|
|1|
|1|
|1|
|1|
|1|
|1|
|1|

 

Can you please clarify why this query is executing or is there any chance that this type of query is executing from our code it self while check for rows count from dataframe.

 

Please provide me your inputs on this.
Bug: Performance issue when Spark SQL connects with TeraData 
Status: Resolved | Priority: Major | Resolution: Done
Created: 09/Nov/22 05:42
Affects Version/s: 2.4.4

We are connecting Tera data from spark SQL with below API

{color:#ff8b00}Dataset<Row> jdbcDF = spark.read().jdbc(connectionUrl, tableQuery, connectionProperties);{color}

We are facing one issue when we execute above logic on large table with million rows every time we are seeing below extra query is executing every time as this resulting performance hit on DB.

This below information we got from DBA. We dont have any logs on SPARK SQL.

SELECT 1 FROM ONE_MILLION_ROWS_TABLE;
|1|
|1|
|1|
|1|
|1|
|1|
|1|
|1|
|1|
|1|

 

Can you please clarify why this query is executing or is there any chance that this type of query is executing from our code it self while check for rows count from dataframe.

 

Please provide me your inputs on this.
Bug: Spark Master OOM when Worker No space left on device
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Nov/22 04:29
Affects Version/s: 2.3.1

Spark master + 1 Spark workers (One is No space left on device)
1.  Submit a app with 2 instances
2.  Stop the good worker 
3.  Spark master launch executor continously
     Bad Spark worker cannot create folder.
     Result in large number of executors kept  in spark master memory
        2022-11-15 22:21:43 INFO  Master:54 - Launching executor app-20221115221952-0016/93441 on worker worker-20221115202400-10.111.1.10-40011
        2022-11-15 22:21:43 INFO  Master:54 - Removing executor app-20221115221952-0016/93441 because it is FAILED
        2022-11-15 22:21:43 INFO  Master:54 - Launching executor app-20221115221952-0016/93442 on worker worker-20221115202400-10.111.1.10-40011
        2022-11-15 22:21:43 INFO  Master:54 - Removing executor app-20221115221952-0016/93442 because it is FAILED
Bug: Fix `SparkSession.builder.config` to support bool
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/22 01:30
Affects Version/s: 3.4.0

Currently, `SparkSession.builder.config` only support string type "true" and "false" to set the configuration, but doesn't work correctly for `True` and `False`.

See [https://github.com/mlflow/mlflow/pull/7307/files#r1019788056] detail.
Bug: Fix new R_LIBS_SITE behavior introduced in R 4.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/22 02:38
Affects Version/s: 3.4.0

R 4.2

{code}
# R
> Sys.getenv("R_LIBS_SITE")
[1] "/usr/local/lib/R/site-library/:/usr/lib/R/site-library:/usr/lib/R/library'"
{code}

{code}
# R --vanilla
> Sys.getenv("R_LIBS_SITE")
[1] "/usr/lib/R/site-library"
{code}

R 4.1

{code}
# R
> Sys.getenv("R_LIBS_SITE")
[1] "/usr/local/lib/R/site-library:/usr/lib/R/site-library:/usr/lib/R/library"
{code}

{code}
# R --vanilla
> Sys.getenv("R_LIBS_SITE")
[1] "/usr/local/lib/R/site-library:/usr/lib/R/site-library:/usr/lib/R/library"
{code}
Bug: Incorrect results or NPE when a literal is reused across distinct aggregations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/22 01:56
Affects Version/s: 3.2.2

This query produces incorrect results:
{noformat}
select a, count(distinct 100) as cnt1, count(distinct b, 100) as cnt2
from values (1, 2), (4, 5) as data(a, b)
group by a;

+---+----+----+
|a  |cnt1|cnt2|
+---+----+----+
|1  |1   |0   |
|4  |1   |0   |
+---+----+----+
{noformat}
The values for {{cnt2}} should be 1 and 1 (not 0 and 0).

If you change the literal used in the first aggregate function, the second aggregate function now works correctly:
{noformat}
select a, count(distinct 101) as cnt1, count(distinct b, 100) as cnt2
from values (1, 2), (4, 5) as data(a, b)
group by a;

+---+----+----+
|a  |cnt1|cnt2|
+---+----+----+
|1  |1   |1   |
|4  |1   |1   |
+---+----+----+
{noformat}
The same bug causes the following query to get a NullPointerException:
{noformat}
select a, count(distinct 1), count_min_sketch(distinct b, 0.5d, 0.5d, 1)
from values (1, 2), (4, 5) as data(a, b)
group by a;
{noformat}
If you change the literal used in the first aggregation, then the query succeeds:
{noformat}
select a, count(distinct 2), count_min_sketch(distinct b, 0.5d, 0.5d, 1)
from values (1, 2), (4, 5) as data(a, b)
group by a;

+---+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|a  |count(DISTINCT 2)|count_min_sketch(DISTINCT b, 0.5, 0.5, 1)                                                                                                                                            |
+---+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|1  |1                |[00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 04 00 00 00 00 5D 8D 6A B9 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00]|
|4  |1                |[00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 04 00 00 00 00 5D 8D 6A B9 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00]|
+---+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
{noformat}
Bug: Koalas.idxmin() is not picking the minimum value from a dataframe, but pandas.idxmin() gives
Status: Open | Priority: Major | Resolution: 
Created: 04/Nov/22 22:24
Affects Version/s: 3.3.1

Hi,
I have a koalas dataframe with age and income and I calculated Zscore on age and income and then norms is calculated using age_zscore and income_zscore(new column name is sq_dist). Then I tried to do an idxmin on the new column, but its not giving the minimum value.
I did the same operations on a Pandas dataframe, but it gives the minimum value .

Please find attached the notebook for step by step operations I performed.
Bug: SQL metrics are missing from Spark UI when AQE for Cached DataFrame is enabled
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Nov/22 06:22
Affects Version/s: 3.4.0

*spark.sql.optimizer.canChangeCachedPlanOutputPartitioning* enables AQE optimizations under InMemoryRelation(IMR) nodes. Following sample query has IMR node on both BroadcastHashJoin legs. However, 
when spark.sql.optimizer.canChangeCachedPlanOutputPartitioning = true, following datas are missed due to lack of final sub-plans (under IMR) submissions (into UI).
{code:java}
- Physical operators' SQLMetrics (before AdaptiveSparkPlan) are missed such as Exchange and HashAggregate on left BHJ leg and HashAggregate on right BHJ leg,
- WSCG blocks are missed on left BHJ leg, 
- AQEShuffleRead node is missed on left BHJ leg. {code}
*Sample to reproduce:*
{code:java}
val spark = SparkSession
    .builder()
    .config("spark.sql.adaptive.enabled", "true")
    .config("spark.sql.optimizer.canChangeCachedPlanOutputPartitioning", "true")
    .master("local[*]")
    .getOrCreate()

import spark.implicits._

// Create 1th DF 
val arr = Seq(
  (1, "Employee_1", "Department_1"),
  (2, "Employee_2", "Department_2"))
val df = arr.toDF("id", "name", "department")
  .filter('id < 3)
  .groupBy('name)
  .count()
df.cache()

// Create 2th DF
val arr2 = Seq((1, "Employee_1", "Department_1"))
val df2 = arr2.toDF("id", "name", "department")
  .filter('id > 0)
  .groupBy('name)
  .count()
df2.cache()

// Trigger query execution
val df3 = df.join(df2, "name")
df3.show() {code}
*DAG before fix:*

*!DAG when AQE=ON and AQECachedDFSupport=ON without fix.png|width=33,height=86!*

*DAG after fix:*

*!DAG when AQE=ON and AQECachedDFSupport=ON with fix.png|width=33,height=82!*
Bug: When RddId exceeds Integer.Max_VALUE, RddId becomes negative
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Nov/22 04:23
Affects Version/s: 3.3.1

The incrementAndGet method of AtomicInteger keeps increasing. When AtomicInteger increases to 2147483647 (Integer. MAX_VALUE) and then adds one, the value of AtomicInteger will become negative - 2147483648 (Integer. MIN_VALUE).

This problem occurs in long running tasks, such as stream tasks.

When BlockManager generates BlockId, BlockId only supports positive rddid, so BlockId generation fails

object BlockId {
val RDD = "rdd_([0-9]+)_([0-9]+)".r
Bug: Introducing a Protobuf serializer for UI data on KV store
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/22 23:02
Affects Version/s: 3.4.0

Introducing Protobuf serializer for KV store, which is 3 times as fast as the default serializer according to end-to-end benchmark against RocksDB.

To move fast and make review easier, the first PR will cover the class `JobDataWrapper` only.
Bug: Getting Count on data frame is giving the performance issue
Status: Open | Priority: Major | Resolution: 
Created: 28/Nov/22 12:38
Affects Version/s: 2.4.4

We are invoking  below query on Teradata 

1) Dataframe<Row> df = spark.format("jdbc"). . . load();

2) int count = df.count();

When we executed the df.count spark internally issuing the below query on teradata which is wasting the lot of CPU on teradata and DBAs are making noise by seeing this query.

 

Query : SELECT 1 FROM (<ONE_MILLION_ROWS_TABLE>)SPARK_SUB_TAB

Response:

1

1

1

1

1

..

1

 

Is this expected behavior from spark or is it bug.
Bug: Spark does not parse timestamp strings when using the IN operator
Status: Open | Priority: Major | Resolution: 
Created: 25/Nov/22 13:36
Affects Version/s: 3.2.1

Likely affects more versions, tested only with 3.2.1.

 

Summary:

Spark will convert a timestamp string to a timestamp when using the equal operator (=), yet won't do this when using the IN operator.

 

Details:

While debugging an issue why we got no results on a query, we found out that when using the equal symbol `=` in the WHERE clause combined with a TimeStampType column that Spark will convert the string to a timestamp and filter.

However, when using the IN operator (our query), it will not do so, and perform a cast to string. We expected the behavior to be similar, or at least that Spark realizes the IN clause operates on a TimeStampType column and thus attempts to convert to timestamp first before falling back to string comparison.

 

*Minimal reproducible example:*

Suppose we have a one-line dataset with the follow contents and schema:

 
{noformat}
+----------------------------+
|starttime                   |
+----------------------------+
|2019-08-11 19:33:05         |
+----------------------------+
root
 |-- starttime: timestamp (nullable = true){noformat}
Then if we fire the following queries, we will not get results for the IN-clause one using a timestamp string with timezone information:

 

 
{code:java}
// Works - Spark casts the argument to a string and the internal representation of the time seems to match it...
singleCol.filter("starttime IN ('2019-08-11 19:33:05')").show();
// Works
singleCol.filter("starttime = '2019-08-11 19:33:05'").show();
// Works
singleCol.filter("starttime = '2019-08-11T19:33:05Z'").show();
// Doesn't work
singleCol.filter("starttime IN ('2019-08-11T19:33:05Z')").show();
//Works
singleCol.filter("starttime IN (to_timestamp('2019-08-11T19:33:05Z'))").show(); {code}
 

We can see from the output that a cast to string is taking place:
{noformat}
[...] isnotnull(starttime#59),(cast(starttime#59 as string) = 2019-08-11 19:33:05){noformat}
Since the = operator does work, it would be consistent if operators such as the IN operator would have similar, consistent behavior.
Bug: Add a physical rule to remove the partialLimitExec node
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 30/Nov/22 13:18
Affects Version/s: 3.4.0

Currently, the planner WindowGroupLimit creates two WindowGroupLimitExec node.

We can add a physical rule to remove the partialLimitExec node, if there is no shuffle between the two nodes (partialLimitExec's outputPartitioning satisfies the finalLimitExec's requiredChildDistribution)
Bug: Add support for distributed deep learning framework
Status: Open | Priority: Major | Resolution: 
Created: 30/Nov/22 19:48
Affects Version/s: 3.3.2

There is a clear trend for deep learning to go from single-machine to distributed to scale/accelerate training. Adding a support for Distributed DL solution on Spark will increase the power for spark and largely simplify the distributed DL workload for the users. 

Currently, [spark-tensorflow-distributor|https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-distributor] provides a solution to run distributed Tensorflow on spark clusters.But there is no such support for distributed PyTorch. 

We want to add a general framework to support both DL frameworks so that we can have a unified interface for distributed DL workload on spark. And it can take the advantages for GPU scheduling on spark and have a better resource management too.
Bug: RocksDB state store WriteBatch doesn't clean up native memory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/22 16:40
Affects Version/s: 3.3.1

The RocksDB state store uses a WriteBatch to hold updates that get written in a single transaction to commit. Somewhat indirectly abort is called after a successful task which calls writeBatch.clear(), but the data for a writeBatch is stored in a std::string in the native code. Not sure why it's stored as a string, but it is. [rocksdb/write_batch.h at main · facebook/rocksdb · GitHub|https://github.com/facebook/rocksdb/blob/main/include/rocksdb/write_batch.h#L491]

writeBatch.clear simply calls rep_.clear() and rep._resize() ([rocksdb/write_batch.cc at main · facebook/rocksdb · GitHub|https://github.com/facebook/rocksdb/blob/main/db/write_batch.cc#L246-L247]), neither of which actually releases the memory built up by a std::string instance. The only way to actually release this memory is to delete the WriteBatch object itself.

Currently, all memory taken by all write batches will remain until the RocksDB state store instance is closed, which never happens during the normal course of operation as all partitions remain loaded on an executor after a task completes.
Bug: Refactor Spark types by introducing physical types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/22 21:18
Affects Version/s: 3.4.0

I am creating this one for Desmond Cheong since he can't signup for an account because of [https://infra.apache.org/blog/jira-public-signup-disabled.html.|https://infra.apache.org/blog/jira-public-signup-disabled.html]
 
His description for this improvement:
The Spark type system currently supports multiple data types with the same physical representation in memory. For example {{DateType}} and {{YearMonthIntervalType}} are both implemented using {{{}IntegerType{}}}. Because of this, operations on data types often involve case matching where multiple data types match to the same effects.To simplify this case matching logic, we can introduce the notion of logical and physical data types where multiple logical data types can be implemented with the same physical data type, then perform case matching on physical data types.Some areas that can utilize this logical/physical type separation are:
 * {{SpecializedGettersReader}} in {{SpecializedGettersReader.java}}
 * {{copy}} in {{ColumnarBatchRow.java}} and {{ColumnarRow.java}}
 * {{getAccessor}} in {{InternalRow.scala}}
 * {{externalDataTypeFor}} in {{RowEncoder.scala}}
 * {{unsafeWriter}} in {{InterpretedUnsafeProjection.scala}}
 * {{getValue}} and {{javaType}} in {{CodeGenerator.scala}}
 * {{doValidate}}  in {{literals.scala}}
Bug: RocksDB State Store WriteBatch doesn't cleanup native memory
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 30/Nov/22 16:41
Affects Version/s: 3.3.1

The RocksDB state store uses a WriteBatch to hold updates that get written in a single transaction to commit. Somewhat indirectly abort is called after a successful task which calls writeBatch.clear(), but the data for a writeBatch is stored in a std::string in the native code. Not sure why it's stored as a string, but it is. [rocksdb/write_batch.h at main · facebook/rocksdb · GitHub|https://github.com/facebook/rocksdb/blob/main/include/rocksdb/write_batch.h#L491]

writeBatch.clear simply calls rep_.clear() and rep._resize() ([rocksdb/write_batch.cc at main · facebook/rocksdb · GitHub|https://github.com/facebook/rocksdb/blob/main/db/write_batch.cc#L246-L247]), neither of which actually releases the memory built up by a std::string instance. The only way to actually release this memory is to delete the WriteBatch object itself.

Currently, all memory taken by all write batches will remain until the RocksDB state store instance is closed, which never happens during the normal course of operation as all partitions remain loaded on an executor after a task completes.
Bug: Support CURRENT_SCHEMA() as alias for CURRENT_DATABASE()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/22 18:33
Affects Version/s: 3.4.0

CURRENT_SCHEMA is the keyword used in thE SQL Standard to refer to the current namespace. It is also supported my multiple other vendors:
PostgreSQL

Redshift

Snowflake
Db2
and others
Bug: Validate scalability improvements due to JDK-8180450 on Netty 4.1.84.Final
Status: Open | Priority: Major | Resolution: 
Created: 29/Nov/22 09:14
Affects Version/s: 3.4.0

[https://netty.io/news/2022/10/11/4-1-84-Final.html] has introduced a bug-fix/improvement to prevent a JDK scalability issue due to type check against interfaces (see [https://github.com/netty/netty/pull/12806] for more details - and [https://redhatperf.github.io/post/type-check-scalability-issue/] for a deeper and wider analysis).

it seems that Spark network make uses of Netty channel duplex handlers on [TransportContext|https://github.com/apache/spark/blob/94de3ca2942bb04852510abccf06df1fa8b2dab3/common/network-common/src/main/java/org/apache/spark/network/TransportContext.java#L201] and it's likely to be affected by it (or better, depending which netty version).

Moreover, I suggest to verify if there are other places where the issue can happen and eventually, report them on https://bugs.openjdk.org/browse/JDK-8180450 as well (if they use type checks in an idiomatic form).
The blog post has linked https://github.com/RedHatPerf/type-pollution-agent to detect dangerous candidate code paths.