av9ash/GitBugs huggingface dataset:
Bug: Support user-defined annotations for OnDemand PVCs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Sep/24 07:38
Affects Version/s: 4.0.0

Currently for on-demand PVCs we cannot add user-defined annotations, user-defined annotations can greatly help to add tags in underlying storage.

For e.g. if we add +k8s-pvc-tagger/tags+ annotation & provide a map like \{"env":"dev"}, the same tags are reflected on underlying storage (for e.g. AWS EBS)
Bug: NPE from org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$asProperties$1
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 20/Sep/24 15:55
Affects Version/s: 3.4.3

NullPointerException occurs in the following stacktrace:
{code:java}
java.lang.NullPointerException
	at java.util.Hashtable.put(Hashtable.java:460)
	at java.util.Properties.setProperty(Properties.java:166)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$asProperties$1(JDBCOptions.scala:55)
	at scala.collection.immutable.Map$Map2.foreach(Map.scala:273)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:55)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcOptionsInWrite.<init>(JDBCOptions.scala:258)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcOptionsInWrite.<init>(JDBCOptions.scala:262)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.saveTableToJDBC(JdbcRelationProvider.scala:54)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:101)
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.$anonfun$sideEffectResult$2(commands.scala:84)
	at org.apache.spark.sql.execution.SparkPlan.runCommandWithAetherOff(SparkPlan.scala:180)
	at org.apache.spark.sql.execution.SparkPlan.runCommandInAetherOrSpark(SparkPlan.scala:191)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.$anonfun$sideEffectResult$1(commands.scala:84) {code}
This occurs because [here,|https://github.com/apache/spark/blob/branch-3.5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommand.scala#L46] the client sends invalid options, where some value in the map is null. We should return appropriate error condition here.
Bug: Fix the error when enable the sparkApplicationSentinel 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Sep/24 04:52
Affects Version/s: kubernetes-operator-0.1.0

When I enable the sparkApplicationSentinel, checking the operator logs with ERROR as below:

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://10.96.0.1:443/apis/spark.apache.org/v1alpha1/namespaces/spark-4/sparkapplications/spark-app-sentinel. Message: SparkApplication.spark.apache.org "spark-app-sentinel" is invalid: spec.runtimeVersions: Required value. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.runtimeVersions, message=Required value, reason=FieldValueRequired, additionalProperties={})], group=spark.apache.org, kind=SparkApplication, name=spark-app-sentinel, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=SparkApplication.spark.apache.org "spark-app-sentinel" is invalid: spec.runtimeVersions: Required value, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
Bug: default value of `spark.sql.warehouse.dir` is not decoded correctly when saving table
Status: Open | Priority: Minor | Resolution: 
Created: 28/Sep/24 16:57
Affects Version/s: 3.5.3

I haven't looked into how _general_ this problem is, but here's a very specific scenario which I ran into last night.

 

When the `{{{}SparkSession{}}}` is created _without_ specifying the config `{{{}spark.sql.warehouse.sql{}}}`, the default value is _cwd/spark-warehouse_ and this path seems URL-encoded when printed via `spark.conf.get('spark.sql.warehouse.dir')`.

So, for instance, if any spaces were present in the path, they will be replaced by "%20".

If this is the case, then the path should be decoded whenever necessary, but it turns out this encoded path is taken literally and consequently spark writes tables to a different location than intended.

 

here's a minimal snippet to reproduce:

```py

{{from pyspark.sql import SparkSession}}

{{spark = SparkSession.builder.getOrCreate()}}
 
{{spark.conf.get('spark.sql.warehouse.dir') # 'file:/Users/user/cwd%20with%20space/spark-warehouse'}}
 
{{df = ...}}
{{df.write.saveAsTable('df') # table will be saved at /Users/user/cwd%20with%20space/spark-warehouse}}
```

 

Interestingly, this doesn't happen if the path is manually specified when creating the session. Even if the path is literally the same as what spark would've taken by-default.

 

```py

{{from pyspark.sql import SparkSession}}

{{{}spark = SparkSession.builder.config('{}}}spark.sql.warehouse.dir', 'spark-warehouse/').getOrCreate()

 
{{spark.conf.get('spark.sql.warehouse.dir') # 'file:/Users/user/cwd with space/spark-warehouse'}}
```

 

The above works fine.

 

PS. plz forgive me if this is supposed to happen by design
Bug: change IOError to OSError
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Sep/24 19:23
Affects Version/s: 4.0.0

OSError is the builtin error type used for exceptions that relate to the operating system.

In Python 3.3, a variety of other exceptions, like WindowsError were aliased to OSError. These aliases remain in place for compatibility with older versions of Python, but may be removed in future versions.

Prefer using OSError directly, as it is more idiomatic and future-proof.

Example

raise IOError
Use instead:


raise OSError

https://docs.astral.sh/ruff/rules/os-error-alias/
https://docs.python.org/3/library/exceptions.html#OSError
Bug: SET CATALOG syntax doesn't support IDENTIFIER 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Sep/24 15:31
Affects Version/s: 4.0.0

Since IDENTIFIER clause was added a lot of gaps remain. One of them is SET CATALOG syntax added in https://issues.apache.org/jira/browse/SPARK-36841

Now {{SET CATALOG IDENTIFIER('test')}} will fall with
{code:java}
[INVALID_SET_SYNTAX] Expected format is 'SET', 'SET key', or 'SET key=value'...{code}
Parser can't match {{SET CATALOG}} with an identifier but can match next "setConfiguration" grammar, what makes the error even more confusing.

It would be nice to support identifier in this context as well.
Bug: Fix to use the exit code of executor container always
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Sep/24 02:12
Affects Version/s: 3.1.1

When deploying Spark pods on Kubernetes with sidecars, the reported executor's exit code may be incorrect.

For example, the reported executor's exit code is 0, but the actual is 52 (OOM).
{code:java}
2024-09-25 02:35:29,383 ERROR TaskSchedulerImpl: org.apache.spark.scheduler.TaskSchedulerImpl.logExecutorLoss(TaskSchedulerImpl.scala:972) - Lost executor 1 on XXXXX: The executor with id 1 exited with exit code 0(success).
  
The API gave the following container statuses:
 
     container name: fluentd
     container image: docker-images-release.XXXXX.com/XXXXX/fluentd:XXXXX
     container state: terminated
     container started at: 2024-09-25T02:32:17Z
     container finished at: 2024-09-25T02:34:52Z
     exit code: 0
     termination reason: Completed
 
     container name: istio-proxy
     container image: docker-images-release.XXXXX.com/XXXXX-istio/proxyv2:XXXXX
     container state: running
     container started at: 2024-09-25T02:32:16Z
 
     container name: spark-kubernetes-executor
     container image: docker-dev-artifactory.XXXXX.com/XXXXX/spark-XXXXX:XXXXX
     container state: terminated
     container started at: 2024-09-25T02:32:17Z
     container finished at: 2024-09-25T02:35:28Z
     exit code: 52
     termination reason: Error {code}
Bug: Increase `spark.test.docker.connectionTimeout` to 10min
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Sep/24 22:49
Affects Version/s: 3.4.3

**MASTER** branch
https://github.com/apache/spark/actions/runs/11045311764/job/30682732260

{code}
[info] OracleIntegrationSuite:
[info] org.apache.spark.sql.jdbc.OracleIntegrationSuite *** ABORTED *** (5 minutes, 17 seconds)
[info]   The code passed to eventually never returned normally. Attempted 298 times over 5.0045005511500005 minutes. Last failure message: ORA-12541: Cannot connect. No listener at host 10.1.0.41 port 41079. (CONNECTION_ID=n9ZWIh+nQn+G9fkwKyoBQA==)
{code}

**branch-3.5**
https://github.com/apache/spark/actions/runs/10939696926/job/30370552237

{code}
[info] MsSqlServerNamespaceSuite:
[info] org.apache.spark.sql.jdbc.v2.MsSqlServerNamespaceSuite *** ABORTED *** (5 minutes, 42 seconds)
[info]   The code passed to eventually never returned normally. Attempted 11 times over 5.487631282400001 minutes. Last failure message: The TCP/IP connection to the host 10.1.0.56, port 35345 has failed. Error: "Connection refused (Connection refused). Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".. (DockerJDBCIntegrationSuite.scala:166)
{code}

**branch-3.4**
https://github.com/apache/spark/actions/runs/10937842509/job/30364658576

{code}
[info] MsSqlServerNamespaceSuite:
[info] org.apache.spark.sql.jdbc.v2.MsSqlServerNamespaceSuite *** ABORTED *** (5 minutes, 42 seconds)
[info]   The code passed to eventually never returned normally. Attempted 11 times over 5.487555645633333 minutes. Last failure message: The TCP/IP connection to the host 10.1.0.153, port 46153 has failed. Error: "Connection refused (Connection refused). Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".. (DockerJDBCIntegrationSuite.scala:166)
{code}
Bug: Fix inaccurate documentation of RuntimeConfig.get
Status: Open | Priority: Minor | Resolution: 
Created: 26/Sep/24 13:32
Affects Version/s: 4.0.0

The existing documentation of `RuntimeConfig.get()` is misleading:
 * `get(key: String)` method will not throw any exception if the key is not set as long as the config entry has a default value, instead, they will just return the `defaultValue` of the `ConfigEntry`. An `NoSuchElementException` will only be thrown if there is no default value for the config entry.
 * `get(key: String, default: String)` method will ignore the `defaultValue` of its `ConfigEntry`, and return the given param `default` if unset. 
 * `getOption(key: String)` method will return the `defaultValue` of its `ConfigEntry` if the config is not set.

 

An example:

This line makes sure the config is not set.
```
scala> spark.conf.unset("spark.sql.session.timeZone") 
```

The following code returns `Etc/UTC`, which doesn't throw any exception.
```
scala> spark.conf.get("spark.sql.session.timeZone") 
res2: String = "Etc/UTC"
```

The following code returns `Europe/Berlin`, ignoring the default value. However, the documentation only says it returns the value, without mentioning ignoring the default value of the entry when the config is not explicitly set.
```
scala> spark.conf.get("spark.sql.session.timeZone", "Europe/Berlin") 
res3: String = "Europe/Berlin"
```

The incorrect documentation is likely to mislead users to weird behaviours if they rely on the documentation.
Bug: Support K8s volume `mount.subPathExpr` and `hostPath` volume `type`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Sep/24 08:15
Affects Version/s: 4.0.0

Spark on K8S supports mounting volumes to driver and executor pods. It supports the {{subPath}} but not the {{subPathExpr}} K8S option, which allows using environment variables for the {{subPath}}. Further, hostPath volumes are always mounted as default type {{""}}. Providing a different hostPath volume type allows for more flexibility.

Add the following configuration options:
- {{spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.subPathExpr}}
- {{spark.kubernetes.executor.volumes.hostPath.[VolumeName].options.type}}
Bug: can mllib functions stringIndexer, ohe be exposed as catalyst expression
Status: Open | Priority: Minor | Resolution: 
Created: 25/Sep/24 05:38
Affects Version/s: 3.5.2

I have been exploring the possibility of exposing the feature transformer via mllib [https://spark.apache.org/docs/latest/ml-features.html] as a catalyst expression to be used within the select query.

for instance, query of 
{code:java}
select string_indexer(col) si_col , binarizer(col2) binarizer_col2 from table{code}
the above sort can be supported via SQL? 

I observed that some of these transformers have been implemented internally as UDFs and others as UDAFs.

Any opinions on this?
Bug: Get rid of unnecessary executions list in SessionHolder
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Sep/24 10:44
Affects Version/s: 4.0.0

ExecutionManager.executions can fully substitute SessionHolder.executions.

Adverse effect.
- interrupt* will take longer if there are many sessions with many executions -> SessionHolder manages a set of operation IDs instead of ExecuteHolders.
Bug: Delay sorting in validateAndMaybeEvolveStateSchema
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Sep/24 14:49
Affects Version/s: 3.4.3

In `{{{}validateAndMaybeEvolveStateSchema`{}}}, existing schema and new schema are sorted by column family name.
The sorting can be delayed until `createSchemaFile` is called.

When computing `{{{}colFamiliesAddedOrRemoved`{}}}, we can use `{{{}toSet`{}}} to compare column families.
Bug: Spark get stuck while evaluating sorted wrong dataframe rdd
Status: Open | Priority: Minor | Resolution: 
Created: 17/Sep/24 13:38
Affects Version/s: 3.5.0

Under the below scenario `sorted.rdd` takes for ever instead of throwing the expected exception because the schema does not fit the data types.

 
{code:java}
import org.apache.spark.sql.Row
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.types._
import scala.util.Try

val data: Seq[Row] = Seq(
  Row(1, "a"),
  Row(2, "b"),
  Row(3, "c")
)

val schema = StructType(Seq(
  StructField("id", StringType),
  StructField("value", StringType)
))

val df = spark.createDataFrame(
  spark.sparkContext.parallelize(data), schema
)

val sorted = df.orderBy(col("value"))

Try(sorted.rdd)
sorted.rdd
{code}
 

 

A less simplified error is happening to us when using Holden Karau Spark Testing Base and as workaround we are forcing an action before the assert, but I guess it is not the expected behaviour.
Bug: E2E Workflow Refactor
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Sep/24 06:27
Affects Version/s: 4.0.0

Current matrix

result in combination

1.28.0, spark-versions, watched-namespace
1.28.0, python, watched-namespace
1.28.0, state-transition, watched-namespace
...

It is costing more time and resources since in every run, the watched-namespace test will be run. It shall be run once after dynamic-mode operator is installed.
Bug: Support user-defined labels for OnDemand PVCs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Sep/24 11:04
Affects Version/s: 4.0.0

Currently when user sets 
volumes.persistentVolumeClaim.[VolumeName].options.claimName=OnDemand
PVCs are created with only 1 label i.e. spark-app-selector = spark.app.id.

Objective of this Jira is to allow support of custom labels for ondemand PVCs
Bug: Detect unused message parameters
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Sep/24 09:56
Affects Version/s: 4.0.0

The passed error message parameters and places holders in message format can not be matched. From the code maintainability perspective, it would be nice to detect such cases while running tests.

For example, the error message format could look like:

{code}
  "CANNOT_UP_CAST_DATATYPE" : {
    "message" : [
      "Cannot up cast <expression> from <sourceType> to <targetType>.",
      "<details>"
    ],
    "sqlState" : "42846"
  },
{code}
 
but the passed message parameters have extra parameter:

{code:scala}
          messageParameters = Map(
            "expression" -> "CAST('aaa' AS LONG)",
            "sourceType" -> "STRING",
            "targetType" -> "LONG",
            "op" -> "CAST", // unused parameter
            "details" -> "implicit cast"
          ))
{code}
Bug: Get rid of coarse-locking in SparkConnectSessionManager
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Sep/24 14:42
Affects Version/s: 4.0.0

Related to https://issues.apache.org/jira/browse/SPARK-49544.
-> This has never caused a real world problem, but we had better fix it in tandem with https://issues.apache.org/jira/browse/SPARK-49544.
Bug: numInputRows is 0 when one of input relation is empty
Status: Open | Priority: Trivial | Resolution: 
Created: 04/Sep/24 20:01
Affects Version/s: 3.4.0

[https://github.com/apache/spark/blob/2ed6c3e511f322c5fd01953736c376a85ff2c687/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala#L481]

 

In case  # relation of logical plan != # relation of physical plan, numInputRows captured as 0 for all sources. However it includes the case that empty relations in union is removed from physical plan by "org.apache.spark.sql.catalyst.optimizer.PropagateEmptyRelation".

 

Repro code:
import org.apache.spark.sql.streaming.Trigger
val df1 = spark.readStream.schema(sch).parquet("/streaming/parquet1")
val df2 = spark.readStream.schema(sch).parquet("/streaming/emptyparquet")
df1.union(df2).writeStream.option("checkpointLocation", "/streaming/checkpoint8").format("parquet").trigger(Trigger.Once()).option("path", "/streaming/result9").start
 
workaround:
spark.conf.set("spark.sql.optimizer.excludedRules","org.apache.spark.sql.catalyst.optimizer.PropagateEmptyRelation")
Bug: Support creating data frame from a Postgres table that contains user-defined array column
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Apr/23 16:36
Affects Version/s: 2.4.0

Spark SQL now doesn’t support creating data frame from a Postgres table that contains user-defined array column. However, it used to allow such type before the Postgres JDBC commit (https://github.com/pgjdbc/pgjdbc/commit/375cb3795c3330f9434cee9353f0791b86125914). The previous behavior was to handle user-defined array column as String.

Given:
 * Postgres table with user-defined array column
 * Function: DataFrameReader.jdbc - https://spark.apache.org/docs/2.4.0/api/java/org/apache/spark/sql/DataFrameReader.html#jdbc-java.lang.String-java.lang.String-java.util.Properties-

Results:
 * Exception “java.sql.SQLException: Unsupported type ARRAY” is thrown

Expectation after the change:
 * Function call succeeds
 * User-defined array is converted as a string in Spark DataFrame

Suggested fix:
 * Update “getCatalystType” function in “PostgresDialect” as
 ** 
{code:java}
val catalystType = toCatalystType(typeName.drop(1), size, scale).map(ArrayType(_))
if (catalystType.isEmpty) Some(StringType) else catalystType{code}
Bug: Restore constructors of exceptions for compatibility in connector API
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 05/Apr/23 19:03
Affects Version/s: 3.4.0

Thanks [~aokolnychyi] for raising the issue as shown below:
{quote}
I have a question about changes to exceptions used in the public connector API, such as NoSuchTableException and TableAlreadyExistsException.

I consider those as part of the public Catalog API (TableCatalog uses them in method definitions). However, it looks like PR #37887 has changed them in an incompatible way. Old constructors accepting Identifier objects got removed. The only way to construct such exceptions is either by passing database and table strings or Scala Seq. Shall we add back old constructors to avoid breaking connectors?
{quote}
We should restore constructors of those exceptions to preserve the compatibility in connector API.
Bug: Spark structuring streaming with kinesis and rocksdb integration
Status: Resolved | Priority: Blocker | Resolution: Invalid
Created: 05/Apr/23 13:08
Affects Version/s: 3.3.1

Hi Spark Team,

We need to read/write the kinesis streams using spark streaming.

 We checked the official documentation - [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]

It does not mention about kinesis connector. Alternative is - [https://github.com/qubole/kinesis-sql] which is not active now.  This is now handed over here - [https://github.com/roncemer/spark-sql-kinesis]

Also according to SPARK-18165 , Spark officially do not have any kinesis connector 

We have few below questions , It would be great if you can answer 
 # Does Spark provides officially any kinesis connector which have readstream/writestream and endorse any connector for production use cases ?  
 # [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html] This documentation does not mention about how to write to kinesis. This method have default dynamodb as checkpoint, can we override it ?
 # We have rocksdb as state store but when we ran application using official  [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html] rocksdb configurations were not effective. Can you please confirm if rocksdb is not applicable in these cases?
 # rocksdb however works with qubole connector , do you have any plan to release kinesis connector?
 # Please help/recommend us for any good stable kinesis connector or some pointers around it
Bug: Parameterized sql() with constants
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Apr/23 08:54
Affects Version/s: 3.4.0

Change Scala/Java/Python APIs to accept any language objects from which would be possible to construct literal columns.

The current implementation the parameterized sql() requires arguments as string values parsed to SQL literal expressions that causes the following issues:
* SQL comments are skipped while parsing, so, some fragments of input might be skipped. For example, 'Europe -- Amsterdam'. In this case, -- Amsterdam is excluded from the input.
* Special chars in string values must be escaped, for instance 'E\'Twaun Moore'