av9ash/GitBugs huggingface dataset:
Bug: Upgrade to PySpark 3.5.0 gives Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/23 08:50
Affects Version/s: 3.5.0

Using PySpark 3.4.1 w/ the following dependencies works fine for reading S3 files:

hadoop-client:3.3.4
hadoop-common:3.3.4
hadoop-aws:3.3.4
aws-java-sdk-bundle:1.12.262

Doing a simple upgrade to PySpark 3.5.0 (which is still using hadoop 3.3.4 AFAIK) results in failing to read the same S3 files:

```
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
	at org.apache.parquet.hadoop.util.HadoopInputFile.fromStatus(HadoopInputFile.java:44)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:76)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$readParquetFootersInParallel$1(ParquetFileFormat.scala:450)
	... 14 more
```
Bug: Specify number of partitions when creating spark dataframe from pandas dataframe
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/23 08:32
Affects Version/s: 3.5.0

When converting a large pandas dataframe to a spark dataframe like so


 
{code:java}
import pandas as pd pdf = pd.DataFrame([{"board_id": "3074457346698037360_0", "file_name": "board-content", "value": "A" * 119251} for i in range(0, 20000)]) spark.createDataFrame(pdf).write.mode("overwrite").format("delta").saveAsTable("catalog.schema.table"){code}
 
 

You can encounter the following error

org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 11:1 was 366405365 bytes, which exceeds max allowed: spark.rpc.message.maxSize (268435456 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values.

 

As far as I can tell spark first converts the pandas dataframe into a python list and then constructs an rdd out of that list. which means that the parallelism is determined by the value of spark.sparkcontext.defaultparallelism and if the pandas dataframe is very large and the number of available cores is low then you end up with very large tasks that exceed the limits imposed on the size of tasks

 

Methods like spark.sparkContext.parallelize allow you to pass in the number of partitions of the resulting dataset. I think having a similar capability when creating a dataframe from a pandas dataframe makes a lot of sense. As right now I think the only workaround I can think of is changing the value of spark.default.parallelism but this is a system wide setting
Bug: Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/23 07:36
Affects Version/s: 4.0.0

{code:java}
@deprecated("`aggregate` is not relevant for sequential collections. Use `foldLeft(z)(seqop)` instead.", "2.13.0")
def aggregate[B](z: => B)(seqop: (B, A) => B, combop: (B, B) => B): B = foldLeft(z)(seqop) {code}
Bug: fix dev/mina get scala 2.12 
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 24/Oct/23 12:15
Affects Version/s: 4.0.0

Now the ci is executing  ./dev/mina will generate an incompatible error with scala2.12. Sorry, I don't know how to fix it
[info] [launcher] getting org.scala-sbt sbt 1.9.3  (this may take some time)...
[info] [launcher] getting Scala 2.12.18 (for sbt)...
Bug: Add toNullable in StructType, MapType and ArrayType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/23 06:53
Affects Version/s: 4.0.0

{{StructType.toNullable}} to return nullable schemas. See https://stackoverflow.com/questions/33193958/change-nullable-property-of-column-in-spark-dataframe as an example.
Bug: spark-sql-kafka-0-10_2.11 - Custom Configuration's for Partitioner not set.
Status: Open | Priority: Major | Resolution: 
Created: 25/Oct/23 17:32
Affects Version/s: 2.4.0

We had a requirement to write Custom org.apache.kafka.clients.producer.Partitioner to use with Kafka Data Source available with package "{{{}spark-sql-kafka-0-10_2.11{}}} "

Ideally, properties set as part of Producer are available to  Partitioner method -

[configure|https://kafka.apache.org/24/javadoc/org/apache/kafka/common/Configurable.html#configure-java.util.Map-]([Map|https://docs.oracle.com/javase/8/docs/api/java/util/Map.html?is-external=true]<[String|https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true],?> configs)

 
 
But, we realized that Custom properties passed as options to Kafka format DataFrameWriter are not available to Partitioner whether we append that property with literal "kafka." or not.

Only, Configs listed on - [https://kafka.apache.org/documentation/#producerconfigs] were passed to Partitioner. But, in some cases it is required to pass custom properties for initialization of Partitioner. 

Thus, there should be provision to set custom properties as options with Kafka Data Source not just producer configs. Otherwise, custom partitioner can't be initialized and implemented as per need.

 

For example - 
_df.write.format("kafka"){color:#57d9a3}.option("Kafka.customproperty1", "value1").option("kafka.partitioner.class", "com.mycustom.ipartitioner"){color}_

..
...
.....
_package com.mycustom;_
_import org.apache.kafka.clients.producer.Partitioner;_
public class _ipartitioner implemets Partitioner{_

_@override_
_public void configure(Map<String,?> configs){_

{color:#FF0000}_system.out.println(configs) //_ _customproperty1 is missing here which should be available._{color}

_}_
_..._
_..._

_}_
Bug: Pyspark  union method throw 'pyspark.sql.utils.IllegalArgumentException: requirement failed'
Status: Open | Priority: Major | Resolution: 
Created: 25/Oct/23 07:01
Affects Version/s: 3.2.1

!image-2023-10-25-15-01-28-896.png|width=1795,height=194!

 

 

The code is 
{code:java}
    output_df = spark.createDataFrame([], output_schema())
    output_df.printSchema()
    print(output_df.count()) {code}
I have checked that the schema is same, `final` is from a sql query.
Bug: Caching SQL UNION of different column data types does not work inside Dataset.union
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/23 00:20
Affects Version/s: 3.3.2

 

Cache SQL UNION of 2 sides with different column data types
{code:java}
scala> spark.sql("select 1 id union select 's2' id").cache()  {code}
Dataset.union does not leverage the cache
{code:java}
scala> spark.sql("select 1 id union select 's2' id").union(spark.sql("select 's3'")).queryExecution.optimizedPlan
res15: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
Union false, false
:- Aggregate [id#109], [id#109]
:  +- Union false, false
:     :- Project [1 AS id#109]
:     :  +- OneRowRelation
:     +- Project [s2 AS id#108]
:        +- OneRowRelation
+- Project [s3 AS s3#111]
   +- OneRowRelation {code}
SQL UNION of the cached SQL UNION does use the cache! Please note `InMemoryRelation` used.
{code:java}
scala> spark.sql("(select 1 id union select 's2' id) union select 's3'").queryExecution.optimizedPlan
res16: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
Aggregate [id#117], [id#117]
+- Union false, false
   :- InMemoryRelation [id#117], StorageLevel(disk, memory, deserialized, 1 replicas)
   :     +- *(4) HashAggregate(keys=[id#100], functions=[], output=[id#100])
   :        +- Exchange hashpartitioning(id#100, 500), ENSURE_REQUIREMENTS, [plan_id=241]
   :           +- *(3) HashAggregate(keys=[id#100], functions=[], output=[id#100])
   :              +- Union
   :                 :- *(1) Project [1 AS id#100]
   :                 :  +- *(1) Scan OneRowRelation[]
   :                 +- *(2) Project [s2 AS id#99]
   :                    +- *(2) Scan OneRowRelation[]
   +- Project [s3 AS s3#116]
      +- OneRowRelation {code}
Bug: Replace `s.c.mutable.MapOps#transform` with `s.c.mutable.MapOps#mapValuesInPlace`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Oct/23 03:54
Affects Version/s: 4.0.0

{code:java}
@deprecated("Use mapValuesInPlace instead", "2.13.0")
@inline final def transform(f: (K, V) => V): this.type = mapValuesInPlace(f) {code}
Bug: FramelessOffsetWindowFunctionFrame fails when ignore nulls and offset > # of rows 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/23 01:06
Affects Version/s: 3.5.0

Failure when function that utilized `FramelessOffsetWindowFunctionFrame` is used with `ignoreNulls = true` and `offset > rowCount`.

e.g. 

```
select x, lead(x, 5) IGNORE NULLS over (order by x) from (select explode(sequence(1, 3)) x)
```
Bug: Use `AccessibleObject#canAccess` instead of `AccessibleObject#isAccessible`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/23 09:31
Affects Version/s: 4.0.0

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala:258:15: method isAccessible in class AccessibleObject is deprecated (since 9)
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.util.UtilsSuite.getFieldValue, origin=java.lang.reflect.AccessibleObject.isAccessible, version=9
[warn]     if (field.isAccessible()) {
[warn]    {code}
Bug: Fix `symbol literal is deprecated`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/23 17:34
Affects Version/s: 4.0.0

For the code `val symbol = 'symbol`, it's a compile warning in Scala 2.13, but it's a compile error in Scala 3.
 - Scala 2.13 

 
{code:java}
Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.8).
Type in expressions for evaluation. Or try :help.
scala> val symbol = 'symbol
                    ^
       warning: symbol literal is deprecated; use Symbol("symbol") instead [quickfixable]
val symbol: Symbol = Symbol(symbol) {code}
 

 
 * Scala 3

 
{code:java}
Welcome to Scala 3.3.1 (17.0.8, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.
                                                                                                                                                                                                                                                    
scala> val symbol = 'symbol
-- Error: ----------------------------------------------------------------------
1 |val symbol = 'symbol
  |             ^
  |symbol literal 'symbol is no longer supported,
  |use a string literal "symbol" or an application Symbol("symbol") instead,
  |or enclose in braces '{symbol} if you want a quoted expression.
  |For now, you can also `import language.deprecated.symbolLiterals` to accept
  |the idiom, but this possibility might no longer be available in the future. {code}
Bug: Expose SQLSTATE in error message
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/23 15:36
Affects Version/s: 4.0.0

When using spark.sql.error.messageFormat in MINIMAL or STANDARD mode the SQLSTATE is exposed;
We want to extend this to PRETTY mode, now that all errors have SQLSTATEs

We propose to trail the SQLSTATE after the text message, so it does not take away from the reading experience of the message, while still being easily found by tooling or humans.
[<errorClass>] <message> SQLSTATE: <state>
<context>

Example:


{{[DIVIDE_BY_ZERO] ** Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22013}}
{{{}== SQL(line 1, position 8){}}}{{{}==
{}}}{{{}SELECT 1/0
{}}}{{       ^^^}}

Other options considered have been:
{{[DIVIDE_BY_ZERO](22013) ** Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. }}
{{{}== SQL(line 1, position 8){}}}{{{}==
{}}}{{{}SELECT 1/0
{}}}{{       ^^^}}


{{and}}

{{{{[DIVIDE_BY_ZERO] ** Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.}}
{{{}== SQL(line 1, position 8){}}}{{{}=={}}}
{{SELECT 1/0}}
{{       ^^^}}
{{{{SQLSTATE: 22013}}}}
}}{{{}{{}}{}}}
Bug: Fix "Auto-application to `()` is deprecated."
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/23 17:16
Affects Version/s: 4.0.0

For the following case, a compile warning will be issued in Scala 2.13：

 
{code:java}
Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.8).
Type in expressions for evaluation. Or try :help.


scala> class Foo {
     |     def isEmpty(): Boolean = true
     |     def isTrue(x: Boolean): Boolean = x
     |   }
class Foo


scala> val foo = new Foo
val foo: Foo = Foo@7061622


scala> val ret = foo.isEmpty
                     ^
       warning: Auto-application to `()` is deprecated. Supply the empty argument list `()` explicitly to invoke method isEmpty,
       or remove the empty argument list from its definition (Java-defined methods are exempt).
       In Scala 3, an unapplied method like this will be eta-expanded into a function. [quickfixable]
val ret: Boolean = true {code}
But for Scala 3, it is a compile error:
{code:java}
Welcome to Scala 3.3.1 (17.0.8, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.
                                                                                                                                                                                                                                                     
scala> class Foo {
     |     def isEmpty(): Boolean = true
     |     def isTrue(x: Boolean): Boolean = x
     |   }
// defined class Foo
                                                                                                                                                                                                                                                     
scala> val foo = new Foo
val foo: Foo = Foo@591f6f83
                                                                                                                                                                                                                                                     
scala> val ret = foo.isEmpty
-- [E100] Syntax Error: --------------------------------------------------------
1 |val ret = foo.isEmpty
  |          ^^^^^^^^^^^
  |          method isEmpty in class Foo must be called with () argument
  |
  | longer explanation available when compiling with `-explain`
1 error found {code}
Bug: Spark SQL returning incorrect values for full outer join on keys with the same name.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/23 03:38
Affects Version/s: 3.4.1

{{The following query gives the wrong results.}}

 

{{WITH people as (}}
{{  SELECT * FROM (VALUES }}
{{    (1, 'Peter'), }}
{{    (2, 'Homer'), }}
{{    (3, 'Ned'),}}
{{    (3, 'Jenny')}}
{{  ) AS Idiots(id, FirstName)}}
{{{}){}}}{{{}, location as ({}}}
{{  SELECT * FROM (VALUES}}
{{    (1, 'sample0'),}}
{{    (1, 'sample1'),}}
{{    (2, 'sample2')  }}
{{  ) as Locations(id, address)}}
{{{}){}}}{{{}SELECT{}}}
{{  *}}
{{FROM}}
{{  people}}
{{FULL OUTER JOIN}}
{{  location}}
{{ON}}
{{  people.id = location.id}}

{{We find the following table:}}
||id: integer||FirstName: string||id: integer||address: string||
|2|Homer|2|sample2|
|null|Ned|null|null|
|null|Jenny|null|null|
|1|Peter|1|sample0|
|1|Peter|1|sample1|

{{But clearly the first `id` column is wrong, the nulls should be 3.}}

If we rename the id column in (only) the person table to pid we get the correct results:
||pid: integer||FirstName: string||id: integer||address: string||
|2|Homer|2|sample2|
|3|Ned|null|null|
|3|Jenny|null|null|
|1|Peter|1|sample0|
|1|Peter|1|sample1|
Bug: Expose DeterministicLevel as a DeveloperApi
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/23 01:56
Affects Version/s: 3.4.0

{{RDD.getOutputDeterministicLevel}} is a {{DeveloperApi}} which users can override to specify the {{DeterministicLevel}} of the {{RDD}}.
Unfortunately, {{DeterministicLevel}} itself is {{private[spark]}}.

Expose {{DeterministicLevel}} to allow users to users this method.
Bug: Assign name to _LEGACY_ERROR_TEMP_2153
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/23 13:00
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2153* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: stackoverflow when executing rule ExtractWindowExpressions
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 19/Oct/23 03:43
Affects Version/s: 3.2.3

I am encountering stackoverflow errors while executing the following test case. I looked at the source code and it is ExtractWindowExpressions not extracting the window correctly and encountering a dead loop at resolveOperatorsDownWithPruning that is causing it.

{code:scala}
 test("agg filter contains window") {
    val src = Seq((1, "b", "c")).toDF("col1", "col2", "col3")
      .withColumn("test",
        expr("count(col1) filter (where min(col1) over(partition by col2 order by col3)>1)"))
    src.show()
  }
{code}

Now my question is this kind of in agg filter (window) is the correct usage? Or should I add a check like spark sql and throw an error "It is not allowed to use window functions inside WHERE clause"?
Bug: Execution fails when there are subqueries in TakeOrderedAndProjectExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/23 03:49
Affects Version/s: 3.5.0

When there are subqueries in TakeOrderedAndProjectExec, the query can throw this exception:

 java.lang.IllegalArgumentException: requirement failed: Subquery subquery#242, [id=#109|#109] has not finished 

This is because TakeOrderedAndProjectExec does not wait for subquery execution.
Bug: Add Matomo analytics to all released docs pages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/23 21:50
Affects Version/s: 3.5.0

Matomo analytics has been added to some pages of the Spark website. Here is Sean's initial PR: [https://github.com/apache/spark-website/pull/479.|https://www.google.com/url?q=https://github.com/apache/spark-website/pull/479&sa=D&source=docs&ust=1696544881650480&usg=AOvVaw11SNfWcd4UJzlO8EJvzdoe]

You can find analytics for Spark website here: https://analytics.apache.org

We need to add this to all API pages. This is very important for us to prioritize documentation improvements and search engine optimization.
Bug: Assign name to _LEGACY_ERROR_TEMP_2152
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/23 08:10
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2152* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Spark job hangs due to task launch thread failed to create
Status: Open | Priority: Major | Resolution: 
Created: 17/Oct/23 08:33
Affects Version/s: 3.1.2

spark job hangs while web ui show there is one task in running stage keep running for multi hours, while other tasks finished in a few minutes 

executor will never report task launch failed info to driver

 

Below is spark task execute thread launch log:

23/10/17 04:45:42 ERROR Inbox: An error happened while processing message in the inbox for Executor
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
        at org.apache.spark.executor.Executor.launchTask(Executor.scala:270)
        at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receive$1.applyOrElse(CoarseGrainedExecutorBackend.scala:173)
        at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
        at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
        at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
        at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
        at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Bug: Introduce a metadata file for streaming stateful operator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/23 17:23
Affects Version/s: 3.5.0

The information to store in the metadata file:
 * operator name (no need to be unique among stateful operators in the query)

 * state store name
 * numColumnsPrefixKey: > 0 if prefix scan is enabled, 0 otherwise

The body of metadata file will be in json format. The metadata file will be versioned just as other streaming metadata file to be future proof.

The metadata file will improve expose more information about the state store, improves debugability and facilitate the development of state related feature such as reading and writing state and state repartitioning.