av9ash/GitBugs huggingface dataset:
Bug: create *permanent* Spark View from DataFrame via PySpark & Scala DataFrame API
Status: Open | Priority: Major | Resolution: 
Created: 12/Jun/23 17:23
Affects Version/s: 3.5.0

currently only *_temporary_ Spark Views* can be created from a DataFrame:
 * [DataFrame.createTempView|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.createTempView.html#pyspark.sql.DataFrame.createTempView]
 * [DataFrame.createOrReplaceTempView|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.createOrReplaceTempView.html#pyspark.sql.DataFrame.createOrReplaceTempView]
 * [DataFrame.createGlobalTempView|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.createGlobalTempView.html#pyspark.sql.DataFrame.createGlobalTempView]
 * [DataFrame.createOrReplaceGlobalTempView|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.createOrReplaceGlobalTempView.html#pyspark.sql.DataFrame.createOrReplaceGlobalTempView]

When a user needs a _*permanent*_ *Spark View* he has to fall back to Spark SQL ({{{}CREATE VIEW AS SELECT...{}}}).

Sometimes it is easier and more readable to specify the desired logic of the view through {_}Scala/PySpark DataFrame API{_}.
Therefore, I'd like to suggest to implement a new PySpark method that allows creating a _*permanent*_ *Spark View* from a DataFrame (e.g. {{{}DataFrame.createOrReplaceView{}}}).

see also:
 * [https://community.databricks.com/s/question/0D53f00001PANVgCAP/is-there-a-way-to-create-a-nontemporary-spark-view-with-pyspark]
 * [https://lists.apache.org/thread/jzkznvt7cfjhmo77w1tlksxkwyvmvvfb]
Bug: Simplify PartitionedFileUtil API a little
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/23 17:42
Affects Version/s: 3.5.0

Utility methods in {{PartitionedFileUtil}} take both {{file: FileStatusWithMetadata}} and {{{}filePath: Path{}}}, even tho the latter can be obtained easily from the former.

This was originally done for performance reasons, to allow callers to pass a memoized {{Path}} and thus avoid the cost of an additional {{FileStatus.getPath}} call.

Now that we have {{{}FileStatusWithMetadata{}}}, we no longer need the redundancy because the new class can capture the path as a {{{}lazy val{}}}.
Bug: Upgrade AWS SDK to v2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/23 21:29
Affects Version/s: 4.0.0

Here is a design doc:
https://docs.google.com/document/d/1nGWbGTqxuFBG2ftfYYXxzrkipINILfWCOwse36yg7Ig/edit?usp=sharing
Bug: Commands for writting (InsertIntoHadoopFsRelationCommand and InsertIntoHiveTable) should log what they are doing
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/23 20:17
Affects Version/s: 3.4.0

Improvements from SPARK-41763 decoupled the execution of create table and data writing commands in a CTAS (see SPARK-41713).

This means that while the code is cleaner with v1 write implementation limited to InsertIntoHadoopFsRelationCommand and InsertIntoHiveTable, the execution of these operations is less clear than it was before. Previously, the command was present in the physical plan (see explain output below):
 
{{== Physical Plan ==}}
{{CommandResult <empty>}}
{{+- Execute CreateHiveTableAsSelectCommand [Database: default, TableName: test_hive_text_table, InsertIntoHiveTable]}}
{{+- *(1) Scan ExistingRDD[...]}}

But in Spark 3.4.0, this output is:

{{== Physical Plan ==}}
{{CommandResult <empty>}}
{{+- Execute CreateHiveTableAsSelectCommand}}
{{+- CreateHiveTableAsSelectCommand [Database: default, TableName: test_hive_text_table]}}
{{+- Project [...]}}
{{+- SubqueryAlias hive_input_table}}
{{+- View (`hive_input_table`, [...])}}
{{+- LogicalRDD [...], false}}

And the write command is now missing. This makes sense since execution is decoupled, but since there is no log output from InsertIntoHiveTable, there is no clear way to fully know that the command actually executed. 

I would propose that either these commands should add a log message at the INFO level that indicates how many rows were written into what table to make easier for a user to know what has happened from the Spark logs. Another option maybe to update the explain output in Spark 3.4 to handle this, but that might be more difficult and make less sense since the operations are now decoupled.
Bug: Add named argument support for SQL functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/23 18:41
Affects Version/s: 3.5.0

Today, there is increasing demand for named argument functions, especially as we continue to introduce longer and longer parameter lists in our SQL functions. In these functions, many arguments could have default values, making it a waste to specify them all even if it is redundant. This is an umbrella ticket to track smaller subtasks which would be completed for implementing this feature.
Bug: Upgrade ORC to 2.0.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/23 20:07
Affects Version/s: 4.0.0

Apache ORC community has the following release cycles which are synchronized with Apache Spark releases.
 * ORC v2.0.0 (next year) for Apache Spark 4.0.x
 * ORC v1.9.0 (this month) for Apache Spark 3.5.x
 * ORC v1.8.x for Apache Spark 3.4.x
 * ORC v1.7.x for Apache Spark 3.3.x
 * ORC v1.6.x for Apache Spark 3.2.x
Bug: SPIP: Stored Procedures API for Catalogs
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/23 17:02
Affects Version/s: 4.0.0

Similar to engines like Trino, it would be great to provide a connector API for stored procedures and the corresponding CALL statements to invoke them.

Such extensions are already used by projects like Hudi and Iceberg. Many custom Delta operations can be represented in the same way, reducing the amount of parser code to maintain.

Proposal is available [here|https://docs.google.com/document/d/1rDcggNl9YNcBECsfgPcoOecHXYZOu29QYFrloo2lPBg].
Bug: Free memory allocated by large vectors when vectors are reset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/23 04:13
Affects Version/s: 3.5.0

When spark reads a data file into a WritableColumnVector, the memory allocated by the WritableColumnVectors is not freed until the VectorizedColumnReader completes.

It will save memory allocation time by reusing the allocated array objects. But it also takes up too many unused memory after the current large vector batch has been read.

Add a memory reserve policy for this scenario which will reuse the allocated array object for small column vectors and free the memory for huge column vectors.

!image-2023-06-29-12-58-12-256.png!!image-2023-06-29-13-03-15-470.png!
Bug: Enable `spark.authenticate` by default in K8s environment
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/23 16:37
Affects Version/s: 4.0.0

Apache Spark supports spark.authenticate and spark.authenticate.secret since 1.0.0.

This issue proposes to set `spark.authenticate=true` simply in K8s environment. There is no other required change because Spark will automatically generate an authentication secret unique to each application for a little improved isolation and security per applications in K8s environment.
Bug: Drop K8s v1.25 and lower version support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/23 20:33
Affects Version/s: 4.0.0

*1. Default K8s Version in Public Cloud environments*

The default K8s versions of public cloud providers are already K8s 1.27+.

- EKS: v1.27 (Default)
- GKE: v1.27 (Stable), v1.27 (Regular), v1.27 (Rapid)

*2. End Of Support*

In addition, K8s 1.25 and olders are going to reach EOL when Apache Spark 4.0.0 arrives on June 2024. K8s 1.26 is also going to reach EOL on June.

|| K8s  ||   AKS   ||   GKE   ||   EKS   ||
| 1.27 | 2024-07 | 2024-08 | 2024-07 |
| 1.26 | 2024-03 | 2024-06 | 2024-06 |
| 1.25 | 2023-12 | 2024-02 | 2024-05 |
| 1.24 | 2023-07 | 2023-10 | 2024-01 |

- [AKS EOL Schedule](https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar)
- [GKE EOL Schedule](https://cloud.google.com/kubernetes-engine/docs/release-schedule)
- [EKS EOL Schedule](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar)
Bug: Post listenerBus events during ExecutePlanRequest
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/23 23:46
Affects Version/s: 3.5.0

Post events SparkListenerConnectOperationStarted, SparkListenerConnectOperationParsed, SparkListenerConnectOperationCanceled,  SparkListenerConnectOperationFailed, SparkListenerConnectOperationFinished, SparkListenerConnectOperationClosed & SparkListenerConnectSessionClosed.

Mirror what is currently available for in HiveThriftServer2EventManager
Bug: Migrating Junit4 to Junit5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/23 02:33
Affects Version/s: 4.0.0

JUnit5 is a powerful and flexible update to the JUnit framework, and it provides a variety of improvements and new features to organize and
describe test cases, as well as help in understanding test results：
 # JUnit 5 leverages features from Java 8 or later, such as lambda functions, making tests more powerful and easier to maintain, but Junit 4 still a Java 7 compatible version
 # JUnit 5 has added some useful new features for describing, organizing, and executing tests. For examples: [Parameterized Tests|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests] and [Conditional Test Execution|https://junit.org/junit5/docs/current/user-guide/#extensions-conditions] may make our test code look simpler, [Parallel Execution|https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution] may make our test faster

 

More importantly, Junit4 is currently an inactive project, which has not released a new version for more than two years
Bug: Find alternative solutions for `com.clearspring.analytics:stream`
Status: Open | Priority: Major | Resolution: 
Created: 29/Jun/23 02:06
Affects Version/s: 4.0.0

The project https://github.com/addthis/stream-lib has been archived and there will be no new maintenance or support in the future. We should look for an alternative solution, perhaps 4.0 is the appropriate time?
Bug: Introduce python breaking change detection
Status: Open | Priority: Major | Resolution: 
Created: 19/Jun/23 09:17
Affects Version/s: 4.0.0

We have breaking change detections for Binary Compatibility and Protobufs, but we don't have one for python.



Authors of [aexpy|https://github.com/StardustDL/aexpy] are willing to help PySpark detecting python breaking changes.
Bug: Optimize BroadcastHashJoin skew when localShuffleReader is disabled
Status: Open | Priority: Major | Resolution: 
Created: 15/Jun/23 07:44
Affects Version/s: 3.5.0

In RemoteShuffleService services such as uniffle and celeborn, it is recommended to disable localShuffleReader by default for better performance. But it may make BroadcastHashJoin skewed, so I want to optimize BroadcastHashJoin skew in OptimizeSkewedJoin when localShuffleReader is disabled.

 

Refer to:

https://github.com/apache/incubator-celeborn#spark-configuration

https://github.com/apache/incubator-uniffle/blob/master/docs/client_guide.md#support-spark-aqe
Bug: Support list-like for binary ops
Status: Open | Priority: Major | Resolution: 
Created: 13/Jun/23 02:31
Affects Version/s: 3.5.0

We should fix the error below:
{code:java}
>>> pser = pd.Series([1, 2, 3, 4, 5, 6], name="x")
>>> psser = ps.from_pandas(pser)
>>> other = [np.nan, 1, 3, 4, np.nan, 6]
>>> psser <= other
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/haejoon.lee/Desktop/git_store/spark/python/pyspark/pandas/base.py", line 412, in __le__
    return self._dtype_op.le(self, other)
  File "/Users/haejoon.lee/Desktop/git_store/spark/python/pyspark/pandas/data_type_ops/num_ops.py", line 242, in le
    _sanitize_list_like(right)
  File "/Users/haejoon.lee/Desktop/git_store/spark/python/pyspark/pandas/data_type_ops/base.py", line 199, in _sanitize_list_like
    raise TypeError("The operation can not be applied to %s." % type(operand).__name__)
TypeError: The operation can not be applied to list.{code}
Bug: Improve parsing of well known wrapper types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/23 15:50
Affects Version/s: 3.4.0

Under `com.google.protobuf`, there are some well known wrapper types for primitives, [namely|https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/wrappers.proto], useful for distinguishing between absence of primitive fields and their default values, as well as for use within `google.protobuf.Any` types. These types are:

{code}
DoubleValue
FloatValue
Int64Value
Uint64Value
Int32Value
Uint32Value
BoolValue
StringValue
BytesValue
{code}

Currently, when we deserialize these from a serialized protobuf into a spark struct, we expand them as if they were normal messages. Concretely, if we have

{code}
syntax = "proto3";

import "google/protobuf/wrappers.proto"

message WktExample {
  google.protobuf.BoolValue bool_val = 1;
  google.protobuf.Int32Value int32_val = 2;
}
{code}

And a message like
{code}
WktExample(true, 100)
{code}

Then the behavior today is to deserialize this as.
{code}
{"bool_val": {"value": true}, "int32_val": {"value": 100}}
{code}

This is quite difficult to work with and not in the spirit of the wrapper type, so it would be nice to deserialize as

{code}
{"bool_val": true, "int32_val": 100}
{code}

This is also the behavior by other popular deserialization libraries, including java protobuf util [Jsonformat|https://github.com/protocolbuffers/protobuf/blob/main/java/util/src/main/java/com/google/protobuf/util/JsonFormat.java#L904-L914] and golangs [jsonpb|https://github.com/gogo/protobuf/blob/master/jsonpb/jsonpb.go#L207-L214].

So for consistency with other libraries and improved usability, I propose we deserialize well known types in this way.
Bug: Add support for EXCEPT in select clause, similar to what databricks provides
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/23 12:22
Affects Version/s: 3.4.0

I'm looking for a way to incorporate the {{select * except(col1, ...)}} clause provided by Databricks into my workflow. I don't use Databricks and would like to introduce this {{select except}} clause either as a spark-package or by contributing a change to Spark.

However, I'm unsure about how to begin this process and would appreciate any guidance from the community.

[https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select.html#examples]

 

Thank you for your assistance.
Bug: Support K8s scheduling gates
Status: Open | Priority: Major | Resolution: 
Created: 20/Jun/23 20:22
Affects Version/s: 4.0.0

[https://kubernetes.io/docs/concepts/scheduling-eviction/pod-scheduling-readiness/]
 - Kubernetes v1.26 [alpha]
 - Kubernetes v1.27 [beta]
 - Kubernetes v1.28 [beta]
Bug: When a task failed and the inferred task for that task is still executing, the number of dynamically scheduled executors will be calculated incorrectly
Status: Open | Priority: Major | Resolution: 
Created: 25/Jun/23 11:14
Affects Version/s: 3.4.1

Assuming a stage has Task 1, with Task 1.0 and a speculative task Task 1.1 running concurrently, the dynamic scheduler calculates the number of executors as 2 (pendingTask: 0, pendingSpeculative: 0, running: 2).

At this point, Task 1.0 fails, and the dynamic scheduler recalculates the number of executors as 2 (pendingTask: 1, pendingSpeculative: 0, running: 1).

Due to the failure of Task 1.0, copyRunning(1) becomes 1. As a result, Task 1 is speculated again and a SparkListenerSpeculativeTaskSubmitted event is triggered. However, the dynamic scheduler's calculation for the number of executors becomes 3 (pendingTask: 1, pendingSpeculative: 1, running: 1), which is obviously not as expected.

Then, Task 1.2 starts, and it is marked as a speculative task. However, the dynamic scheduler still calculates the number of executors as 3 (pendingTask: 1, pendingSpeculative: 1, running: 1), which again is not as expected.
Bug: Add some, bool_or,bool_and,every to Scala and Python
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 01:24
Affects Version/s: 3.5.0

Add following functions:

* -any-
* some
* bool_or
* bool_and
* every

to:

* Scala API
* Python API
* Spark Connect Scala Client
* Spark Connect Python Client
Bug: SPIP: Python Data Source API
Status: Open | Priority: Major | Resolution: 
Created: 16/Jun/23 04:05
Affects Version/s: 4.0.0

This proposal aims to introduce a simple API in Python for Data Sources. The idea is to enable Python developers to create data sources without having to learn Scala or deal with the complexities of the current data source APIs. The goal is to make a Python-based API that is simple and easy to use, thus making Spark more accessible to the wider Python developer community. This proposed approach is based on the recently introduced Python user-defined table functions (SPARK-43797) with extensions to support data sources.

{*}SPIP{*}: [https://docs.google.com/document/d/1oYrCKEKHzznljYfJO4kx5K_Npcgt1Slyfph3NEk7JRU/edit?usp=sharing]
Bug: Use the value of spark.eventLog.compression.codec set by user when write compact file
Status: Open | Priority: Major | Resolution: 
Created: 07/Jun/23 07:27
Affects Version/s: 3.4.0

Currently, if enable rolling log in SHS, only {{originalFilePath}} is used to determine the path of compact file.
{code:java}
override val logPath: String = originalFilePath.toUri.toString + EventLogFileWriter.COMPACTED
{code}
If the user set {{spark.eventLog.compression.codec}} in sparkConf and it is different from the default value of spark conf, when the log compact logic is triggered, the old event log file will be compacted and use the default value of spark conf.
{code:java}
protected val compressionCodec =
    if (shouldCompress) {
      Some(CompressionCodec.createCodec(sparkConf, sparkConf.get(EVENT_LOG_COMPRESSION_CODEC)))
    } else {
      None
    }

private[history] val compressionCodecName = compressionCodec.map { c =>
    CompressionCodec.getShortName(c.getClass.getName)
  }
{code}
However, The compression codec used by EventLogFileReader to read log is split from the log path, this will lead to EventLogFileReader can not read the compacted log file normally.
{code:java}
def codecName(log: Path): Option[String] = {
    // Compression codec is encoded as an extension, e.g. app_123.lzf
    // Since we sanitize the app ID to not include periods, it is safe to split on it
    val logName = log.getName.stripSuffix(COMPACTED).stripSuffix(IN_PROGRESS)
    logName.split("\\.").tail.lastOption
  }
{code}
So we should override the {{shouldCompress}} and {{compressionCodec}} variable in class {{{}CompactedEventLogFileWriter{}}}, use the compression codec set by the user.
Bug: Add extra per-rule validation for optimization rewrites.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/23 21:47
Affects Version/s: 3.4.0

Adds per-rule validation checks for the following:

1.  aggregate expressions in Aggregate plans are valid.
2. Grouping key types in Aggregate plans cannot by of type Map. 
3. No dangling references have been generated.

This is validation is by default enabled for all tests or selectively using the spark.sql.planChangeValidation=true flag.
Bug: CSV Table Read Error with CharType(length) column
Status: Open | Priority: Major | Resolution: 
Created: 12/Jun/23 04:09
Affects Version/s: 3.4.0

Problem:
 # read a CSV format table
 # table has a `CharType(length)` column
 # read table failed with Exception:  `org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 36.0 failed 4 times, most recent failure: Lost task 0.3 in stage 36.0 (TID 72) (10.113.9.208 executor 11): java.lang.IllegalArgumentException: requirement failed: requiredSchema (struct<name:string,age:int,job:string>) should be the subset of dataSchema (struct<name:string,age:int,job:string>).`

 

reproduce with official image:
 # {{docker run -it apache/spark:v3.4.0 /opt/spark/bin/spark-sql}}
 # {{CREATE TABLE csv_bug (name STRING, age INT, job CHAR(4)) USING CSV OPTIONS ('header' = 'true', 'sep' = ';') LOCATION "/opt/spark/examples/src/main/resources/people.csv";}}
 # SELECT * FROM csv_bug;
 # ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.IllegalArgumentException: requirement failed: requiredSchema (struct<name:string,age:int,job:string>) should be the subset of dataSchema (struct<name:string,age:int,job:string>).
Bug: Generate operator does not update reference set properly
Status: Open | Priority: Major | Resolution: 
Created: 16/Jun/23 19:18
Affects Version/s: 3.5.0

Before 
```
== Optimized Logical Plan ==
Project [col1#2, col2#19]
+- Generate replicaterows(sum#17L, col1#2, col2#3), [2], false, [col1#2, col2#3]
   +- Filter (isnotnull(sum#17L) AND (sum#17L > 0))
      +- Aggregate [col1#2, col2#19], [col1#2, col2#19, sum(vcol#14L) AS sum#17L]
         +- Union false, false
            :- Aggregate [col1#2], [1 AS vcol#14L, col1#2, first(col2#3, false) AS col2#19]
            :  +- LogicalRDD [col1#2, col2#3], false
            +- Project [-1 AS vcol#15L, col1#8, col2#9]
               +- LogicalRDD [col1#8, col2#9], false
```

Couldn't find col2#3 in [col1#2,col2#19,sum#17L]

after 
```
== Optimized Logical Plan ==
Project [col1#2, col2#19]
+- Generate replicaterows(sum#17L, col1#2, col2#19), [2], false, [col1#2, col2#19]
   +- Filter (isnotnull(sum#17L) AND (sum#17L > 0))
      +- Aggregate [col1#2, col2#19], [col1#2, col2#19, sum(vcol#14L) AS sum#17L]
         +- Union false, false
            :- Aggregate [col1#2], [1 AS vcol#14L, col1#2, first(col2#3, false) AS col2#19]
            :  +- LogicalRDD [col1#2, col2#3], false
            +- Project [-1 AS vcol#15L, col1#8, col2#9]
               +- LogicalRDD [col1#8, col2#9], false
```