av9ash/GitBugs huggingface dataset:
Bug: Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/24 13:19
Affects Version/s: 2.0.0

+*Background:*+
PostgresDialect.getTableExistsQuery is using LIMIT 1 query to check the table existence in the database by overriding the default JdbcDialect.getTableExistsQuery which has WHERE 1 = 0.

+*Issue:*+
Due to LIMIT 1 query pattern, we are seeing high number of shared locks in the PostgreSQL installations where there are many partitions under a table that's being written to. Hence resorting to the default JdbcDialect which does WHERE 1 = 0 is proven to be more optimal as it doesn't scan any of the partitions and effectively checks for table existence.

The SELECT 1 FROM table LIMIT 1 query can indeed be heavier in certain scenarios, especially with partitioned tables or tables with a lot of data, as it may take shared locks on all partitions or involve more planner and execution time to determine the quickest way to get a single row.

On the other hand, SELECT 1 FROM table WHERE 1=0 doesn't actually try to read any data due to the always false WHERE condition. This makes it a lighter operation, as it typically only involves checking the table's metadata to validate the table's existence without taking locks on the table's data or partitions.

So, considering performance and minimizing locks, SELECT 1 FROM table WHERE 1=0 would be a better choice if we're strictly looking to check for a table's existence and want to avoid potentially heavier operations like taking shared locks on partitions.
Bug: get_json_object flattens wildcard queries that match a single value
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/24 18:47
Affects Version/s: 3.4.1

I think this impacts all versions of {{{}get_json_object{}}}, but I am not 100% sure.

The unit test for [$.store.book[*].reader|https://github.com/apache/spark/blob/39f8e1a5953b5897f893151d24dc585a80c0c8a0/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala#L142-L146] verifies that the output of this query is a single level JSON array, but when I put the same JSON and JSON path into [http://jsonpath.com/] I get a result with multiple levels of nesting. It looks like Apache Spark tries to flatten lists for {{[*]}} matches when there is only a single element that matches.
{code:java}
scala> Seq("""[{"a":"A"},{"b":"B"}]""","""[{"a":"A"},{"a":"B"}]""").toDF("jsonStr").selectExpr("""get_json_object(jsonStr,"$[*].a")""").show(false)
+--------------------------------+
|get_json_object(jsonStr, $[*].a)|
+--------------------------------+
|"A"                             |
|["A","B"]                       |
+--------------------------------+ {code}
But this has problems in that I no longer have a consistent schema returned, even if the input schema is known to be consistent. For example if I wanted to know how many elements matched this query I could wrap it in a {{json_array_length}} but that does not work in the generic case.
{code:java}
scala> Seq("""[{"a":"A"},{"b":"B"}]""","""[{"a":"A"},{"a":"B"}]""").toDF("jsonStr").selectExpr("""json_array_length(get_json_object(jsonStr,"$[*].a"))""").show(false)
+---------------------------------------------------+
|json_array_length(get_json_object(jsonStr, $[*].a))|
+---------------------------------------------------+
|null                                               |
|2                                                  |
+---------------------------------------------------+ {code}
If the value returned might be a JSON array, and then I would get a number, but it is wrong.
{code:java}
scala> Seq("""[{"a":[1,2,3,4,5]},{"b":"B"}]""","""[{"a":[1,2,3,4,5]},{"a":[1,2,3,4,5]}]""").toDF("jsonStr").selectExpr("""json_array_length(get_json_object(jsonStr,"$[*].a"))""").show(false)
+---------------------------------------------------+
|json_array_length(get_json_object(jsonStr, $[*].a))|
+---------------------------------------------------+
|5                                                  |
|2                                                  |
+---------------------------------------------------+ {code}
Bug: Incorrect count() of a dataframe loaded from CSV datasource
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/24 14:14
Affects Version/s: 4.0.0

The example below portraits the issue:
{code:java}
>>> df=spark.read.option("multiline", "true").option("header", "true").option("escape", '"').csv("es-939111-data.csv")
>>> df.count()
4
>>> df.cache()
DataFrame[jobID: string, Name: string, City: string, Active: string]
>>> df.count()
5{code}
Bug: Support ListState in Arbitrary State API v2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jan/24 01:33
Affects Version/s: 4.0.0

As part of Arbitrary State API v2 ([https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig),] we need to support ListState. This task encounters adding support for ListState in Scala.
Bug: Add a check for stateful operator change for streaming
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/24 18:00
Affects Version/s: 4.0.0

Currently user will get a misleading error as org.apache.spark.sql.execution.streaming.state.StateSchemaNotCompatible if restarting query in the same checkpoint location and changing their stateful operator. We need to catches such errors and throws a new error with informative message.

After physical planning, before execution phase, we will read from state metadata with the current operator id to fetch operator name of committed batch with the same operator id. If operator name does not match, throws the error.
Bug: Replace UnsupportedOperationException by SparkUnsupportedOperationException in catalyst
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jan/24 09:39
Affects Version/s: 4.0.0

Replace all UnsupportedOperationException by SparkUnsupportedOperationException in Catalyst code base, and introduce new legacy error classes with the _LEGACY_ERROR_TEMP_ prefix.
Bug: Replace UnsupportedOperationException by SparkUnsupportedOperationException in sql/core
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/24 18:58
Affects Version/s: 4.0.0

Replace all UnsupportedOperationException by SparkUnsupportedOperationException in sql/core code base, and introduce new legacy error classes with the _LEGACY_ERROR_TEMP_ prefix.
Bug: CSV fails on a column with default and without enforcing schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/24 08:49
Affects Version/s: 4.0.0

When we create a table using CSV on an existing file with a header and:
 - a column has an default +
 - enforceSchema is false - taking into account CSV header

then query a column with a default.

The example below shows the issue:
{code:sql}
CREATE TABLE IF NOT EXISTS products (
  product_id INT,
  name STRING,
  price FLOAT default 0.0,
  quantity INT default 0
)
USING CSV
OPTIONS (
  header 'true',
  inferSchema 'false',
  enforceSchema 'false',
  path '/Users/maximgekk/tmp/products.csv'
);
{code}
The CSV file products.csv:
{code:java}
product_id,name,price,quantity
1,Apple,0.50,100
2,Banana,0.25,200
3,Orange,0.75,50
{code}
The query fails:
{code:sql}
spark-sql (default)> SELECT price FROM products;
24/01/28 11:43:09 ERROR Executor: Exception in task 0.0 in stage 8.0 (TID 6)
java.lang.IllegalArgumentException: Number of column in CSV header is not equal to number of fields in the schema:
 Header length: 4, schema size: 1
CSV file: file:///Users/maximgekk/tmp/products.csv
{code}
Bug: Extend SELECT * support outside of select list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/24 20:29
Affects Version/s: 4.0.0

Traditionally * is confined to thr select list and there to the top level of expressions.
Spark does, in an undocumented fashion support * in the SELECT list for function argument list.
Here we want to expand upon this capability by adding the WHERE clause (Filter) as well as a couple of more scenarios such as row value constructors and IN operator.
Bug: Add support for a custom prefix for Union type fields in Avro when enableStableIdentifiersForUnionType is enabled 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jan/24 02:25
Affects Version/s: 4.0.0

{{enableStableIdentifiersForUnionType}} allows to enable stable identifiers in Avro. We need to add another config to allow configuring the custom prefix used in stable identifiers. Currently the value is {{{}member_{}}}, e.g. member_int, member_string, but we should be able to change to any value or even leave empty.
Bug: Remove inline scripts from UI descriptions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/24 17:59
Affects Version/s: 3.4.1

Users can inject inline scripts (e.g. {{onclick}} or {{onmouseover}} handlers) in the UI job and stage descriptions.

The UI already has precaution to treat, e.g., {{<script>}} tags as plain-text. But that doesn't extend to inline scripts.

Example:

{code:title=Bad job descriptions}
scala> sc.setJobDescription("""<a href="/link" onmouseover="alert('oops');">onmouseover</a>""")

scala> spark.sql("SELECT 1").show()
...

scala> sc.setJobDescription("""<a href="/link" onclick="alert('oops');">onclick</a>""")

scala> spark.sql("SELECT 1").show()
...
{code}

 !Screenshot 2024-01-29 at 09.06.34.png|width=600!
Bug: Could not initialize class org.apache.spark.unsafe.array.ByteArrayMethods error in JDK 21
Status: Open | Priority: Major | Resolution: 
Created: 29/Jan/24 20:32
Affects Version/s: 3.3.1

Trying to run spark on JDK 21, and we're getting this exception
{code:java}
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.unsafe.array.ByteArrayMethods
	at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes$lzycompute(MemoryManager.scala:264)
	at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes(MemoryManager.scala:254)
	at org.apache.spark.memory.MemoryManager.$anonfun$pageSizeBytes$1(MemoryManager.scala:273)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.memory.MemoryManager.<init>(MemoryManager.scala:273)
	at org.apache.spark.memory.UnifiedMemoryManager.<init>(UnifiedMemoryManager.scala:58)
	at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:207)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:320)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at 
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "skir-0"]
	at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:56)
	at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes$lzycompute(MemoryManager.scala:264)
	at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes(MemoryManager.scala:254)
	at org.apache.spark.memory.MemoryManager.$anonfun$pageSizeBytes$1(MemoryManager.scala:273)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.memory.MemoryManager.<init>(MemoryManager.scala:273)
	at org.apache.spark.memory.UnifiedMemoryManager.<init>(UnifiedMemoryManager.scala:58)
	at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:207)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:320)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at  {code}
Bug: PySpark spark.streams should not recreate new StreamingQueryManager
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/24 06:35
Affects Version/s: 4.0.0

In Scala, there is only one streaming query manager for one spark session:

```

scala> spark.streams

val *res0*: *org.apache.spark.sql.streaming.StreamingQueryManager* = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val *res1*: *org.apache.spark.sql.streaming.StreamingQueryManager* = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val *res2*: *org.apache.spark.sql.streaming.StreamingQueryManager* = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val *res3*: *org.apache.spark.sql.streaming.StreamingQueryManager* = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

```

 

In Python, this is currently false:

```

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7c10>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f71f0>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7be0>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7c40>

```

 

Python should align scala behavior.
Bug: Credentials with https url not working for --jars, --files, --archives & --py-files options on spark-submit command
Status: Open | Priority: Major | Resolution: 
Created: 25/Jan/24 09:27
Affects Version/s: 3.3.3

We are trying to run the spark application by pointing the dependent files as well the main pyspark script from secure webserver

We are looking for solution to pass the dependencies as well as pysaprk script from webserver.

we have tried deploying the spark application from webserver to k8s cluster without username and password and it worked, but when tried with username/password we are facing "Exception in thread "{*}main" java.io.IOException: Server returned HTTP response code: 401 for URL: https://username:password@domain.com/application/pysparkjob.py{*}"

*Working  options on spark-submit:*
spark-submit ......

--repositories https://username:password@domain.com/repo1/repo

--jars https://domain.com/jars/runtime.jar \

--files https://domain.com/files/query.sql \

--py-files [https://domain.com/pythonlib/pythonlib.zip] \

https://domain.com/app1/pysparkapp.py

Note: only repositories option works with username and password

*Spark-submit using https url with username/password not working:*

spark-submit ......

--jars https://username:password@domain.com/jars/runtime.jar \

--files https://username:password@domain.com/files/query.sql \

--py-files https://username:password@domain.com[/pythonlib/pythonlib.zip|https://domain.com/pythonlib/pythonlib.zip] \

https://username:password@domain.com/app1/pysparkapp.py

 

Error :

25/01/23 09:19:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 401 for URL: https://username:password@domain.com/repository/spark-artifacts/pysparkdemo/1.0/pysparkdemo-1.0.tgz
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2000)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
        at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:809)
        at org.apache.spark.util.DependencyUtils$.downloadFile(DependencyUtils.scala:264)
        at org.apache.spark.util.DependencyUtils$.$anonfun$downloadFileList$2(DependencyUtils.scala:233)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
Bug: RocksDB versionID Mismatch in SST files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/24 20:28
Affects Version/s: 3.4.1

We need to ensure that the correct SST files are used on executor during RocksDB load as per mapping in metadata.zip. With current implementation, its possible that the executor uses a SST file (with a different UUID) from a older version which is not the exact file mapped in the metadata.zip. This can cause version Id mismatch errors while loading RocksDB leading to streaming query failures.

Few scenarios in which such a situation can occur are:

**Scenario 1 - Distributed file system does not support overwrite functionality**
 # A task T1 on executor A commits Rocks Db snapshot for version X.
 # Another task T2 on executor A loads version X-1, and tries to commit X. During commit, SST files are copied but metadata file is not overwritten.
 # Task T3 is scheduled on A, this task reuses previously loaded X (loaded in (2) above) and commits X+1.
 # Task T4 is scheduled on A again for state store version X. The executor deletes SST files corresponding to commit X+1, downloads the metadata for version X (which was committed in task T1), and loads RocksDB. This would fail because the metadata in (1) is not compatible with SST files in (2).

 

**Scenario 2 - Multiple older State versions have different DFS files for a particular SST file.**


In the current logic, we look at all the versions older than X to find if a local SST file can be reused. The reuse logic only ensures that the local SST file was present in any of the previous version. However, its possible that 2 different older versions had a different SST file (`0001-uuid1.sst` and `0001-uuid2.sst`) uploaded on DFS. These SST files will have the same local name (with UUID truncated) and size, but are not compatible due to different RocksDB Version Ids. We need to ensure that the correct SST file (as per UUID) is picked as mentioned in the metadata.zip.
Bug: Github Action: TypeError: Cannot read properties of undefined (reading 'head_sha') 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/24 09:06
Affects Version/s: 4.0.0

{code:java}
// code placeholder

Ref: SPARK-46772
121SHA: 04008bc2797e67c718417c87f526918556a98ebf
122TypeError: Cannot read properties of undefined (reading 'head_sha')
123    at eval (eval at callAsyncFunction (/home/runner/work/_actions/actions/github-script/v7/dist/index.js:35424:16), <anonymous>:81:22)
124    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
125    at async main (/home/runner/work/_actions/actions/github-script/v7/dist/index.js:35522:20)
126Error: Unhandled error: TypeError: Cannot read properties of undefined (reading 'head_sha') {code}
Bug: Add DAYNAME function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jan/24 12:12
Affects Version/s: 4.0.0

Add function DAYNAME which returns three-letter abbreviation of a given date/timestamp to various spark apis (scala, python, r, connect)

Similar to [46515|https://issues.apache.org/jira/projects/SPARK/issues/SPARK-46515?filter=allissues]


cc: [~mitkedb]
Bug: Incorrect results due to inferred predicate from checkpoint with subquery 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/24 14:00
Affects Version/s: 3.5.0

Spark can produce incorrect results when using a checkpointed DataFrame with a filter containing a scalar subquery. This subquery is included in the constraints of the resulting LogicalRDD, and may then be propagated as a filter when joining with the checkpointed DataFrame. This causes the subquery to be evaluated twice: once during checkpointing and once while evaluating the query. These two subquery evaluations may return different results, e.g. when the subquery contains a limit with an underspecified sort order.
Bug: Do not treat exit 5 as a test failure in Python testing script
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/24 00:49
Affects Version/s: 4.0.0

{code}
========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.12']
Will test the following Python modules: ['pyspark-core', 'pyspark-streaming', 'pyspark-errors']
python3.12 python_implementation is CPython
python3.12 version is: Python 3.12.1
Starting test(python3.12): pyspark.streaming.tests.test_context (temp output: /__w/spark/spark/python/target/8674ed86-36bd-47d1-863b-abb0405557f6/python3.12__pyspark.streaming.tests.test_context__umu69c3v.log)
Finished test(python3.12): pyspark.streaming.tests.test_context (12s)
Starting test(python3.12): pyspark.streaming.tests.test_dstream (temp output: /__w/spark/spark/python/target/847eb56b-3c5f-49ab-8a83-3326bb96bc5d/python3.12__pyspark.streaming.tests.test_dstream__rorhk0lc.log)
Finished test(python3.12): pyspark.streaming.tests.test_dstream (102s)
Starting test(python3.12): pyspark.streaming.tests.test_kinesis (temp output: /__w/spark/spark/python/target/78f23c83-c24d-4fa1-abbd-edb90f48dff1/python3.12__pyspark.streaming.tests.test_kinesis__q5l1pv0h.log)
test_kinesis_stream (pyspark.streaming.tests.test_kinesis.KinesisStreamTests.test_kinesis_stream) ... skipped "Skipping all Kinesis Python tests as environmental variable 'ENABLE_KINESIS_TESTS' was not set."
test_kinesis_stream_api (pyspark.streaming.tests.test_kinesis.KinesisStreamTests.test_kinesis_stream_api) ... skipped "Skipping all Kinesis Python tests as environmental variable 'ENABLE_KINESIS_TESTS' was not set."

----------------------------------------------------------------------
Ran 0 tests in 0.000s

NO TESTS RAN (skipped=2)

Had test failures in pyspark.streaming.tests.test_kinesis with python3.12; see logs.
Error:  running /__w/spark/spark/python/run-tests --modules=pyspark-core,pyspark-streaming,pyspark-errors --parallelism=1 --python-executables=python3.12 ; received return code 255
Error: Process completed with exit code 19.
{code}

Scheduled job fails because of exit 5, see https://github.com/pytest-dev/pytest/issues/2393. This isn't a test failure.
Bug: Grouping by subquery with a cached relation can fail
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/24 21:07
Affects Version/s: 3.4.2

Example:
{noformat}
create or replace temp view data(c1, c2) as values
(1, 2),
(1, 3),
(3, 7),
(4, 5);

cache table data;

select c1, (select count(*) from data d1 where d1.c1 = d2.c1), count(c2) from data d2 group by all;
{noformat}
It fails with the following error:
{noformat}
[INTERNAL_ERROR] Couldn't find count(1)#163L in [c1#78,_groupingexpression#149L,count(1)#82L] SQLSTATE: XX000
org.apache.spark.SparkException: [INTERNAL_ERROR] Couldn't find count(1)#163L in [c1#78,_groupingexpression#149L,count(1)#82L] SQLSTATE: XX000
{noformat}
If you don't cache the view, the query succeeds.

Note, in 3.4.2 and 3.5.0 the issue happens only with cached tables, not cached views. I think that's because cached views were not getting properly deduplicated in those versions.
Bug: Support Java `Set` in JavaTypeInference
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/24 08:11
Affects Version/s: 4.0.0

Scala Set (scala.collection.Set) is supported in ScalaReflection so users can encode Scala Set in Dataset. But Java Set is not supported in bean encoder (i.e., JavaTypeInference). This feature inconsistency makes Java users cannot use Set like Scala users do.