av9ash/GitBugs huggingface dataset:
Bug: Load `spark-defaults.conf` when passing configurations to `spark-submit` through `--properties-file`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/24 20:50
Affects Version/s: 3.5.1

Currently, when user pass configurations to {{spark-submit.sh}} via {{--properties-file}}, the {{spark-defaults.conf}} will be completely ignored. This poses issues for some people, for instance, those using [Spark on K8S operator from kubeflow|https://github.com/kubeflow/spark-operator]. See related issues: 
- https://github.com/kubeflow/spark-operator/issues/1183
- https://github.com/kubeflow/spark-operator/issues/1321
Bug: Support additional PyArrow Table column types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/24 13:20
Affects Version/s: 3.5.1

SPARK-48220 adds support for passing a PyArrow Table to {{{}createDataFrame(){}}}, but there are a few PyArrow column types that are not yet supported:
 * fixed-size binary
 * fixed-size list
 * large list
Bug: [M1] Multiple results API - sqlScript()
Status: Open | Priority: Major | Resolution: 
Created: 21/May/24 15:06
Affects Version/s: 4.0.0

For now:
 * Write an API proposal
 ** The API itself should be fine, but we need to figure out what the result set should look like, i.e. in what format we return multiple DataFrames.
 ** The result set should be compatible with CALL and EXECUTE IMMEDIATE as well.
 * Figure out how the API will propagate down the Spark Connect stack (depends on SPARK-48452 investigation)

 

Probably to be separated into multiple subtasks in the future.
Bug: [M1] Performance benchmark
Status: Open | Priority: Major | Resolution: 
Created: 29/May/24 09:11
Affects Version/s: 4.0.0

Performance parity is officially an M2 requirement, but by the end of M0 I think we should start doing some perf benchmarks to figure out where do we stand in the beginning and if we need to change something right from the start before we get to work on a more complex stuff.
Bug: [M1] Support for PRINT/TRACE statement
Status: Open | Priority: Major | Resolution: 
Created: 29/May/24 08:54
Affects Version/s: 4.0.0

This is not defined in Ref Spec[,|https://docs.google.com/document/d/1cpSuR3KxRuTSJ4ZMQ73FJ4_-hjouNNU2zfI4vri6yhs/edit#heading=h.4cz970y1mk93],] but during POC we figured out that it might be useful.

Still need to figure out the details when we get to it, because the propagation to the client and UI on the client side might not be trivial, but this needs further investigation.
Bug: [M0] Testing and operational readiness
Status: Open | Priority: Major | Resolution: 
Created: 29/May/24 09:16
Affects Version/s: 4.0.0

We are basically doing this as we are developing the feature. This work item should serve as a checkpoint by the end of M0 to figure out if we have covered everything.

 

Testing is very clearly defined by itself.

For the operational readiness part, we are still to figure out what exactly we can do in the case of SQL scripting. It's a really straightforward feature and public documentation should serve well enough for most of the issues we might encounter. But, we should probably think about:
 * Some KPI indicators.
 * Telemetry.
 * Something else?
Bug: Simplify conditionals with predicate branches
Status: Open | Priority: Major | Resolution: 
Created: 20/May/24 14:53
Affects Version/s: 3.4.3

The catalyst optimizer has many optimizations for {{CaseWhen}} and {{If}} expressions, that can eliminate branches entirely or replace them with Boolean logic. There are additional "always false" conditionals that could be eliminated entirely. It would also be possible to replace conditionals with Boolean logic where the {{if-branch}} and {{else-branch}} are themselves predicates. The primary motivation would be to push more filters to the datasource.

For example:
{code:java}
Filter(If(GreaterThan(a, 2), false, LessThanOrEqual(b <= 4))){code}
is equivalent to
{code:java}
# a not nullable
Filter(And(LessThanOrEqual(a, 2), LessThanOrEqual(b, 4))

# a nullable
Filter(And(Not(EqualNotSafe(GreaterThan(a, 2), true), LessThanOrEqual(b, 4)))) {code}
Within a filter the nullability handling is admittedly less important since the expression evaluating to null would be semantically equivalent to false, but the original conditional may have been intentionally written to not return null when {{a}} may be null.
Bug: Cleanup mapIdToMapIndex on mapoutput unregister
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/May/24 02:22
Affects Version/s: 3.5.0

There is only one valid mapstatus for the same {{mapIndex}} at the same time in Spark. {{mapIdToMapIndex}} should also follows the same rule to avoid chaos.

 

The issue leads to shuffle fetch failure and the job failure in the end. It happens this way:
 # Stage A computes the partition P0 by task t1 (TID, a.k.a mapId) on executor e1
 # Executor Y starts deommission
 # Executor Y reports false-positve lost to driver during its decommission
 # Stage B reuse the shuffle dependency with Stage A, and computes the partition P0 again by task t2 on executor e2
 # When task t2 finishes, we see two items ((t1 -> P0), (t2 -> P0)) for the same paritition in {{mapIdToMapIndex}} but only one item (mapStatuses(P0)=MapStatus(t2, e2)) in {{{}mapStatuses{}}}.
 # Executor Y starts to migrate task t1's mapstatus (to executor e3 for example) and call {{updateMapOutput}} on driver. Regarding to 5), we'd use mapId (i.e., t1) to get mapIndex (i.e., P0) and use P0 to get task t2's mapstatus.

// updateMapOutputval mapIndex = mapIdToMapIndex.get(mapId)val mapStatusOpt = mapIndex.map(mapStatuses(_)).flatMap(Option(_))
  # Task t2's mapstatus's location then would be updated to executor e3 but it's indeed still located on executor e2. This finally leads to the fetch failure in the end.
Bug: use addAll instead of add function  in TaskMetrics  to accelerate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/24 13:04
Affects Version/s: 3.5.0

In the fromAccumulators method of TaskMetrics，we should use `
tm._externalAccums.addAll` instead of `tm._externalAccums.add`, as _externalAccums is a instance of CopyOnWriteArrayList
Bug: Foldable propagation replace foldable column should use origin column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/24 01:07
Affects Version/s: 3.0.3

column name will be change by `FoldablePropagation` in optimizer

befor optimizer:

```shell

'Project ['x, 'y, 'z]
+- 'Project ['a AS x#112, str AS Y#113, 'b AS z#114]
   +- LocalRelation <empty>, [a#0, b#1]

```

after optimizer:

```shell

Project [x#112, str AS Y#113, z#114]
+- Project [a#0 AS x#112, str AS Y#113, b#1 AS z#114]
   +- LocalRelation <empty>, [a#0, b#1]

```

column name `y` will be replace to 'Y'
Bug: Check state store provider class before invoking the constructor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/24 00:34
Affects Version/s: 4.0.0

We should restrict that only classes [extending|https://github.com/databricks/runtime/blob/1440e77ab54c40981066c22ec759bdafc0683e76/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala#L73] {{StateStoreProvider}} can be constructed to prevent customer from instantiating arbitrary class of objects.
Bug: In KafkaOffsetReader, partition mismatch should not be an assertion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/24 22:21
Affects Version/s: 4.0.0

In KafkaOffsetReader, we assert startOffsets have the same topic partition list as assigned. However, if the user changes topic partition while the query is running, they will see the assertion. Instead, they should see an exception.
Bug: Use static regex Pattern instances in common/utils JavaUtils
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/24 21:53
Affects Version/s: 4.0.0

Some methods in JavaUtils.java are recompiling regexes on every invocation; we should instead store a single cached Pattern.

This is a minor perf. issue that I spotted in the context of other profiling. Not a huge bottleneck in the grand scheme of things, but simple and straightforward to fix.
Bug: Allow passing PyArrow Table to createDataFrame()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/24 13:34
Affects Version/s: 3.5.1

SPARK-47365 added support for returning a Spark DataFrame as a PyArrow Table.

It would be nice if we could also go in the opposite direction, enabling users to create a Spark DataFrame from a PyArrow Table by passing the PyArrow Table to {{spark.createDataFrame()}}.
Bug: OptimizeOneRowPlan should not be effective for streaming DataFrame
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/24 02:51
Affects Version/s: 3.4.4

This bug is introduce from Apache Spark 3.3.0.

OptimizeOneRowPlan aggressively rewrites operators or removes operators if the rule figures out that the operator has a chance to be optimized with the stats that there will be max 1 row in the input.

This is problematic for streaming, because we aren't seeing the whole data but a part of data in the current microbatch and optimizer is not aware of this.

There are various viable approaches to deal with, but maybe fixing the rule to disable this effectively with streaming DataFrame would have least effect.

(There is a separate wider effort to achieve better stability between QO and streaming. Since it would take a considerable time, we still need point fixes during the time.)
Bug: Avoid no-op empty relation propagation in AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/24 21:11
Affects Version/s: 4.0.0

We should avoid no-op empty relation propagation in AQE: if we convert an empty QueryStageExec to empty relation, it will further wrapped into a new query stage and execute -> produce empty result -> empty relation propagation again. This issue is currently not exposed because AQE will try to reuse shuffle.
Bug: Fix map value extraction when map contains collated strings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/24 11:45
Affects Version/s: 4.0.0

Following queries return unexpected results:
{code:java}
select collation(map('a', 'b' collate utf8_binary_lcase)['a']);
select collation(element_at(map('a', 'b' collate utf8_binary_lcase), 'a'));{code}
Both return UTF8_BINARY instead of UTF8_BINARY_LCASE.
Bug: Replace NullPointerExceptions with proper error classes in AssertNotNull expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/24 17:13
Affects Version/s: 4.0.0

[Code location here|https://github.com/apache/spark/blob/f5d9b809881552c0e1b5af72b2a32caa25018eb3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala#L1929]
Bug: Allow passing iterator of PyArrow RecordBatches to createDataFrame()
Status: Open | Priority: Major | Resolution: 
Created: 30/May/24 21:02
Affects Version/s: 3.5.1

As a follow-up to SPARK-48220:

For larger data, it would be nice to be able to pass an iterator of PyArrow RecordBatches to {{{}createDataFrame(){}}}.
Bug: Do not forward predicates on collated columns to file readers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/24 14:00
Affects Version/s: 4.0.0

SPARK-47657 allows to push filters on collated columns to file sources that support it. If such filters are pushed to file sources, those file sources must not push those filters to the actual file readers (i.e. parquet or csv readers), because there is no guarantee that those support collations.

With this task, we are widening filters on collations to be AlwaysTrue when we translate filters for file sources.
Bug: Support Jdbc datasource custom data partitioning
Status: Open | Priority: Major | Resolution: 
Created: 29/May/24 03:12
Affects Version/s: 4.0.0

"partitionColumn, lowerBound, upperBound" is not an efficient table partitioning scheme for some databases, The amount of data in each partition is consistent. Such as: Oracle has more efficient data table partitioning，

[https://github.com/apache/sqoop/blob/f8beae32a067d72bf9ed6e903b041ad347ca5491/src/java/org/apache/sqoop/manager/oracle/OraOopOracleQueries.java#L335C51-L335C76]
Bug: Fail to Create Table with VOID columns in ORC Format with CTAS
Status: Open | Priority: Major | Resolution: 
Created: 29/May/24 04:08
Affects Version/s: 3.2.2

When I executed Spark SQL:

*CREATE TABLE t1 STORED AS ORC AS SELECT NULL AS id, 'abc' AS name*

 

I got the following error stack:

org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: Unknown primitive type VOID at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:111) at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:245) at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:94) at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:376) at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectBase.run(CreateHiveTableAsSelectCommand.scala:69) at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectBase.run$(CreateHiveTableAsSelectCommand.scala:41) at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:113) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111) at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:97) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:788) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:93)
Bug: Mismatched Row Count for Tables in Text Format Created by CTAS
Status: Open | Priority: Major | Resolution: 
Created: 29/May/24 02:31
Affects Version/s: 2.3.2

After I executed the following Spark SQL to create Hive table with default file format {*}text{*}:

 

*CREATE TABLE t1 AS SELECT '123' AS id, 'abc\ndef' AS name*

 

I got the following result from {*}SELECT * FROM t1{*}:

 

id     name

123  abc

def   null

 

How to make the result match the input without changing the file format ?
Bug: Don't inline UDFs with non-cheap children in CollapseProject
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/24 21:41
Affects Version/s: 3.5.1

Because UDFs (and certain other expressions) are considered cheap by CollapseProject.isCheap, they are inlined and potentially duplicated (which is ok, because rules like ExtractPythonUDFs will de-duplicate them). However, if the UDFs contain other non-cheap expressions, those will also be duplicated and can potentially cause performance regressions.
Bug: Replaces pyspark-connect to pyspark_connect for its output name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/24 03:26
Affects Version/s: 4.0.0

The issue is at setuptools from 69.X.X.

It replaces dash in package name to underscore (`pyspark_connect-4.0.0.dev1.tar.gz` vs `pyspark-connect-4.0.0.dev1.tar.gz`)

https://github.com/pypa/setuptools/issues/4214
Bug: Filesystems do not load with spark.jars.packages configuration
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 24/May/24 20:40
Affects Version/s: 3.5.1

When we use spark.jars.packages configuration parameter in Python SparkSession Builder (Pyspark), it appears that the filesystems are not loaded when session starts. Because of this, Spark fails to read file from Google Cloud Storage (GCS) bucket (with GCS Connector). 

I tested this with different packages so it does not appear specific to a particular package. I will attach the sample code and debug logs.
Bug: Upgrade commons-cli to 1.8.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/24 03:05
Affects Version/s: 4.0.0

* [https://commons.apache.org/proper/commons-cli/changes-report.html#a1.7.0]
 * [https://commons.apache.org/proper/commons-cli/changes-report.html#a1.8.0|https://commons.apache.org/proper/commons-cli/changes-report.html#a1.7.8]
Bug: Driver and Executor support merge and run in a single jvm
Status: Open | Priority: Major | Resolution: 
Created: 24/May/24 01:30
Affects Version/s: 4.0.0

Spark is used in data integration scenarios (such as reading data from mysql and writing data to other data sources), and in many cases can run tasks in a single concurrency. The Driver and Executor consume resources separately. If Driver and Executor support merging, especially when running on the cloud. Can save calculation cost.
Bug: Spark Connect Go Session Type cannot be used in complex code
Status: Open | Priority: Major | Resolution: 
Created: 23/May/24 07:41
Affects Version/s: 3.5.1

The return type of a Spark Session builder is an unexported type.  This means that code cannot pass around a sql.sparkSession to perform more complex tasks (for instance, a routine that sets up a set of temporary views for later processing on).

 
spark, err := sql.SparkSession.Builder.Remote(*remote).Build()
Bug: DateFormatClass (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/24 09:51
Affects Version/s: 4.0.0

Enable collation support for the *DateFormatClass* built-in function in Spark. First confirm what is the expected behaviour for this expression when given collated strings, and then move on to implementation and testing. You will find this expression in the *datetimeExpressions.scala* file, and it should be considered a pass-through function with respect to collation awareness. Implement the corresponding E2E SQL tests (CollationSQLExpressionsSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *DateFormatClass* expression so that it supports all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Ascii, Chr, Base64, UnBase64, Decode, StringDecode, Encode, ToBinary, FormatNumber, Sentences).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for string [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Allow schema parameter of createDataFrame() to be length-1 list or tuple of StructType
Status: Closed | Priority: Major | Resolution: Won't Fix
Created: 21/May/24 13:01
Affects Version/s: 3.5.1

Currently in PySpark (both Classic and Connect), if a user passes a length-1 list or tuple of {{StructType}} as the {{schema}} argument to {{{}createDataFrame{}}}, PySpark raises an unhelpful error message.

Unfortunately it is easy for this to happen. For example if a user leaves a trailing comma at the end of the line that defines the {{{}StructType{}}}.

Add some simple code to {{createDataFrame}} to handle this case more gracefully
Bug: Fix lint-scala for scalafmt to detect properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/24 07:22
Affects Version/s: 4.0.0

{code}
./build/mvn \
    -Pscala-2.13 \
    scalafmt:format \
    -Dscalafmt.skip=false \
    -Dscalafmt.validateOnly=true \
    -Dscalafmt.changedOnly=false \
    -pl connector/connect/common \
    -pl connector/connect/server \
    -pl connector/connect/client/jvm
{code}

fails as below:

{code}
[INFO] Scalafmt results: 1 of 36 were unformatted
[INFO] Details:
[INFO] - Requires formatting: ConnectProtoUtils.scala
[INFO] - Formatted: UdfUtils.scala
[INFO] - Formatted: DataTypeProtoConverter.scala
[INFO] - Formatted: ConnectCommon.scala
[INFO] - Formatted: ProtoUtils.scala
[INFO] - Formatted: Abbreviator.scala
[INFO] - Formatted: ProtoDataTypes.scala
[INFO] - Formatted: LiteralValueProtoConverter.scala
[INFO] - Formatted: InvalidPlanInput.scala
[INFO] - Formatted: ForeachWriterPacket.scala
[INFO] - Formatted: StreamingListenerPacket.scala
[INFO] - Formatted: StorageLevelProtoConverter.scala
[INFO] - Formatted: UdfPacket.scala
[INFO] - Formatted: ClassFinder.scala
[INFO] - Formatted: SparkConnectClient.scala
[INFO] - Formatted: GrpcRetryHandler.scala
[INFO] - Formatted: GrpcExceptionConverter.scala
[INFO] - Formatted: ArrowEncoderUtils.scala
[INFO] - Formatted: ScalaCollectionUtils.scala
[INFO] - Formatted: ArrowDeserializer.scala
[INFO] - Formatted: ArrowVectorReader.scala
[INFO] - Formatted: ArrowSerializer.scala
[INFO] - Formatted: ConcatenatingArrowStreamReader.scala
[INFO] - Formatted: RetryPolicy.scala
[INFO] - Formatted: SparkConnectStubState.scala
[INFO] - Formatted: ArtifactManager.scala
[INFO] - Formatted: SparkResult.scala
[INFO] - Formatted: RetriesExceeded.scala
[INFO] - Formatted: CloseableIterator.scala
[INFO] - Formatted: package.scala
[INFO] - Formatted: ExecutePlanResponseReattachableIterator.scala
[INFO] - Formatted: ResponseValidator.scala
[INFO] - Formatted: SparkConnectClientParser.scala
[INFO] - Formatted: CustomSparkConnectStub.scala
[INFO] - Formatted: CustomSparkConnectBlockingStub.scala
[INFO] - Formatted: TestUDFs.scala
{code}

This is because the output format has changed due to scalafmt version upgrade.
Bug: FileStreamSource shouldn't double cache files for availableNow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/24 22:20
Affects Version/s: 3.5.1

FileStreamSource loads and saves all files at initialization for Trigger.AvailableNow. However files will also be cached in unreadFiles, which is a waste and causes issues identified in https://issues.apache.org/jira/browse/SPARK-44924 for streams that are reading more than 10k files per batch. We should always skip using the unreadFiles cache when using available now trigger, as there is no need for it anyway.
Bug: Limit the maximum number of dynamic partitions
Status: Open | Priority: Major | Resolution: 
Created: 21/May/24 15:34
Affects Version/s: 4.0.0

This issues https://issues.apache.org/jira/browse/SPARK-37217  

‘Assuming that 1001 partitions are written, the data of 10001 partitions will be deleted first, but because {{hive.exec.max.dynamic.partitions}} is 1000 by default, loadDynamicPartitions will fail at this time, but the data of 1001 partitions has been deleted.

So we can check whether the number of dynamic partitions is greater than {{hive.exec.max.dynamic.partitions}} before deleting, it should fail quickly at this time.’

 

Can this be done to make it more comprehensive.


If a task on the execution node has already generated partitions that exceed hive. exec. max. dynamic. partitions, the task should be stopped at the execution node because the cost of generating data is high.


If the executing node does not generate a partition exceeding hive.axec.max.dynamic.partitions for a certain task, it is still determined through the driver.