av9ash/GitBugs huggingface dataset:
Bug: OVR transform fix potential column conflict
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Feb/21 10:08
Affects Version/s: 3.2.0

{code:java}
import org.apache.spark.ml.classification._val df = spark.read.format("libsvm").load("/d0/Dev/Opensource/spark/data/mllib/sample_multiclass_classification_data.txt").withColumn("probability", lit(0.0))val classifier = new LogisticRegression().setMaxIter(1).setTol(1E-6).setFitIntercept(true)
val ovr = new OneVsRest().setClassifier(classifier)
val ovrm = ovr.fit(df)
ovrm.transform(df)
java.lang.IllegalArgumentException: requirement failed: Column probability already exists.
  at scala.Predef$.require(Predef.scala:281)
  at org.apache.spark.ml.util.SchemaUtils$.appendColumn(SchemaUtils.scala:106)
  at org.apache.spark.ml.util.SchemaUtils$.appendColumn(SchemaUtils.scala:96)
  at org.apache.spark.ml.classification.ProbabilisticClassifierParams.validateAndTransformSchema(ProbabilisticClassifier.scala:38)
  at org.apache.spark.ml.classification.ProbabilisticClassifierParams.validateAndTransformSchema$(ProbabilisticClassifier.scala:33)
  at org.apache.spark.ml.classification.LogisticRegressionModel.org$apache$spark$ml$classification$LogisticRegressionParams$$super$validateAndTransformSchema(LogisticRegression.scala:917)
  at org.apache.spark.ml.classification.LogisticRegressionParams.validateAndTransformSchema(LogisticRegression.scala:268)
  at org.apache.spark.ml.classification.LogisticRegressionParams.validateAndTransformSchema$(LogisticRegression.scala:255)
  at org.apache.spark.ml.classification.LogisticRegressionModel.validateAndTransformSchema(LogisticRegression.scala:917)
  at org.apache.spark.ml.PredictionModel.transformSchema(Predictor.scala:222)
  at org.apache.spark.ml.classification.ClassificationModel.transformSchema(Classifier.scala:182)
  at org.apache.spark.ml.classification.ProbabilisticClassificationModel.transformSchema(ProbabilisticClassifier.scala:88)
  at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:71)
  at org.apache.spark.ml.classification.ProbabilisticClassificationModel.transform(ProbabilisticClassifier.scala:107)
  at org.apache.spark.ml.classification.OneVsRestModel.$anonfun$transform$4(OneVsRest.scala:215)
  at scala.collection.IndexedSeqOptimized.foldLeft(IndexedSeqOptimized.scala:60)
  at scala.collection.IndexedSeqOptimized.foldLeft$(IndexedSeqOptimized.scala:68)
  at scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:198)
  at org.apache.spark.ml.classification.OneVsRestModel.transform(OneVsRest.scala:203)
  ... 49 elided {code}
Bug: replace withTimeZone defined in OracleIntegrationSuite with DateTimeTestUtils.withDefaultTimeZone
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Feb/21 02:19
Affects Version/s: 3.2.0

In OracleIntegrationSuite, withTimeZone method is defined and used only in it to change the default timezone.
On the other hand, withDefaultTimeZone method is defined in DateTimeTestUtils as a utility method and it is semantically the same as withTimeZone.
So might be better to replace withTimeZone with withDefaultTimeZone.
Bug: Map JDBC SQL TIME type to TimestampType with time portion fixed regardless of timezone
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Feb/21 11:08
Affects Version/s: 3.2.0

Due to user-experience (confusing to Spark users - java.sql.Time using milliseconds vs Spark using microseconds; and user losing useful functions like hour(), minute(), etc on the column), we have decided to revert back to use TimestampType but this time we will enforce the hour to be consistently across system timezone (via offset manipulation)

Full Discussion with Wenchen Fan [~cloud_fan] regarding this ticket is here https://github.com/apache/spark/pull/30902#discussion_r569186823

Related issues: [SPARK-33888|https://issues.apache.org/jira/browse/SPARK-33888]
Bug: TakeOrderedAndProjectExec avoid shuffle if input rdd has single partition
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Feb/21 05:53
Affects Version/s: 3.2.0

existing impl collects local topKs to a final partition by a ShuffleExchangeExec,

when the input is small and has only one partition, this shuffle can be skipped.
Bug: Push down limit for LEFT SEMI and LEFT ANTI join
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Feb/21 03:26
Affects Version/s: 3.1.0

I found out during code review of [https://github.com/apache/spark/pull/31567|https://github.com/apache/spark/pull/31567,]( [https://github.com/apache/spark/pull/31567#discussion_r577379572] ), where we can push down limit to the left side of LEFT SEMI and LEFT ANTI join, if the join condition is empty.

Why it's safe to push down limit:

The semantics of LEFT SEMI join without condition:

(1). if right side is non-empty, output all rows from left side.

(2). if right side is empty, output nothing.

 

The semantics of LEFT ANTI join without condition:

(1). if right side is non-empty, output nothing.

(2). if right side is empty, output all rows from left side.

 

With the semantics of output all rows from left side or nothing (all or nothing), it's safe to push down limit to left side.

NOTE: LEFT SEMI / LEFT ANTI join with non-empty condition is not safe for limit push down, because output can be a portion of left side rows.

 

Physical operator for LEFT SEMI / LEFT ANTI join without condition - [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala#L200-L204] .
Bug: ArrayIndexOutOfBoundsException when select in different case
Status: Resolved | Priority: Trivial | Resolution: Cannot Reproduce
Created: 14/Feb/21 12:07
Affects Version/s: 3.0.1

h5. Actual behavior:

Select column with different case after remapping fail with ArrayIndexOutOfBoundsException.
h5. Expected behavior:

Spark shouldn't fail with ArrayIndexOutOfBoundsException.
 Spark is case insensetive by default, so select should return selected column.
h5. Test case:
{code:java}
case class User(aA: String, bb: String)
// ...
val user = User("John", "Doe")

val ds = Seq(user).toDS().map(identity)

ds.select("aa").show(false)
{code}
h5. Additional notes:

Test case is reproducible with Spark 3.0.1. There are no errors with Spark 2.4.7.

I belive problem could be solved by changing filter in `SchemaPruning#pruneDataSchema` from this:
{code:java}
val dataSchemaFieldNames = dataSchema.fieldNames.toSet
val mergedDataSchema =
  StructType(mergedSchema.filter(f => dataSchemaFieldNames.contains(f.name)))
{code}
to this:
{code:java}
val dataSchemaFieldNames = dataSchema.fieldNames.map(_.toLowerCase).toSet
val mergedDataSchema =
  StructType(mergedSchema.filter(f => dataSchemaFieldNames.contains(f.name.toLowerCase)))
{code}
Bug: Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available
Status: Resolved | Priority: Trivial | Resolution: Won't Fix
Created: 13/Feb/21 09:56
Affects Version/s: 3.0.0

Took us a while to figure out how to fix this with pyspark this might save a few people a few hours...

 

The documentation describes vaguely how to fix the issue, by setting a parameter but without an actual working example. 

With the given PySpark example it should hold enough information to set this in other scenarios as well.

 

 

Kept the change to the docs as small as possible. 
h3. What changes were proposed in this pull request?

doc update, see title
h3. Why are the changes needed?

save people time figuring out how to resolve it
h3. Does this PR introduce _any_ user-facing change?

no
h3. How was this patch tested?

no code changes
Bug: KMeansSummary class is omitted from PySpark documentation
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 13/Feb/21 04:43
Affects Version/s: 2.4.7

`KMeansSummary` is missing from `__all__` in clustering.py, Sphinx omits it from emitted documentation, and the class is invisible when imported by other modules.
Bug: Add documentation for full outer and left semi stream-stream join
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Feb/21 08:07
Affects Version/s: 3.1.0

Per discussion in https://issues.apache.org/jira/browse/SPARK-32883?focusedCommentId=17285057&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17285057, we should add documentation for added new features of full outer and left semi joins into SS programming guide.
Bug: Clean up unused code for code simplifications
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Feb/21 11:59
Affects Version/s: 3.0.0

Currently, we pass the default value `EmptyRow` to method `checkEvaluation` in the StringExpressionsSuite, but the default value of the 'checkEvaluation' method parameter is the `emptyRow`.

We can clean the parameter for Code Simplifications.

 

example:

*before:*
{code:java}
def testConcat(inputs: String*): Unit = {
  val expected = if (inputs.contains(null)) null else inputs.mkString
  checkEvaluation(Concat(inputs.map(Literal.create(_, StringType))), expected, EmptyRow)
}{code}
*after:*
{code:java}
def testConcat(inputs: String*): Unit = {
  val expected = if (inputs.contains(null)) null else inputs.mkString
  checkEvaluation(Concat(inputs.map(Literal.create(_, StringType))), expected)
}{code}
Bug: Use standard methods to extract keys or values from a Map.
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 05/Feb/21 06:46
Affects Version/s: 3.2.0

For keys：
*before* 

{code:scala}
map.map(_._1)
{code}

*after*

{code:java}
map.keys
{code}

For values:
{code:scala}
map.map(_._2)
{code}

*after*

{code:java}
map.values
{code}
Bug: Spark images are based of Python 3.8 which is soon EOL
Status: Closed | Priority: Blocker | Resolution: Not A Problem
Created: 02/May/24 15:29
Affects Version/s: 3.4.0

Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based on Python 3.8 as of now.

I am proposing to use Python 3.10 as default. Let me know if I can pick this up to make the changes in [spark-docker|[https://github.com/apache/spark-docker]]
Bug: Unable to write MLPipeline to blob storage using .option attribute
Status: Open | Priority: Critical | Resolution: 
Created: 26/May/24 17:40
Affects Version/s: 3.4.3

I am trying to write mllib pipeline with a series of stages set in it to azure blob storage giving relevant write parameters, but it still complains of `fs.azure.account.key` not being found in the configuration.

Sharing the code.
{code:java}
val spark = SparkSession.builder().appName("main").master("local[4]").getOrCreate()

import spark.implicits._

val df = spark.createDataFrame(Seq(
  (0L, "a b c d e spark"),
  (1L, "b d")
)).toDF("id", "text") 

val si = new StringIndexer().setInputCol("text").setOutputCol("IND_text")
val pipelinee = new Pipeline().setStages(Array(si))
val pipelineModel = pipelinee.fit(df)
val path = BLOB_STORAGE_PATH

pipelineModel.write
.option("spark.hadoop.fs.azure.account.key.<account_name>.dfs.core.windows.net",  "__").option("fs.azure.account.key.<account_name>.dfs.core.windows.net", "__").option("fs.azure.account.oauth2.client.endpoint.<account_name>.dfs.core.windows.net", "__").option("fs.azure.account.oauth2.client.id.<account_name>.dfs.core.windows.net", "__").option("fs.azure.account.auth.type.<account_name>.dfs.core.windows.net","__").option("fs.azure.account.oauth2.client.secret.<account_name>.dfs.core.windows.net", "__").option("fs.azure.account.oauth.provider.type.<account_name>.dfs.core.windows.net", "__")
.save(path){code}
 

The error that i get is 
{code:java}
 Failure to initialize configuration
Caused by: InvalidConfigurationValueException: Invalid configuration value detected for fs.azure.account.key
at org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:51)
    at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:548)
    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1449){code}
This shows that even though the key,value of 
{code:java}
spark.hadoop.fs.azure.account.key.<account_name>.dfs.core.windows.net {code}
is being sent via option param, but is not being set internally.

 

while this works only if i explicitly set the values in the
{code:java}
spark.conf.set(key,value) {code}
which might be problematic for a multi-tenant solution, which can be using the same spark context.

one other observation is 
{code:java}
df.write.option(key1,value1).option(key2,value2).save(path)  {code}
fails with same key error while,
{code:java}
map = Map(key1->value1, key2->value2)  
df.write.options(map).save(path) {code}
works..

 

Help required on: Similar to how dataframes `options`
{code:java}
df.write.options(Map<key,value>) {code}
 helps to set the configuration, the *.option(key1, value1)* should also work to write to azure blob storage.
Bug: NettyServer doesn't shutdown if SparkContext initialize failed
Status: Open | Priority: Critical | Resolution: 
Created: 20/May/24 03:06
Affects Version/s: 3.1.3

When obtaining a SparkContext instance using SparkContext.getOrCreate(), if an exception occurs during initialization (such as using incorrect Spark parameters, e.g., spark.executor.memory=1 without units), the RpcServer started during this period will not be shut down, resulting in the port being occupied indefinitely.

The action to close the RpcServer happens in _env.stop(), where rpcEnv.shutdown() is executed, but this action only occurs when _env != null (SparkContext.scala:2106, version 3.1.3). However, the error occurs during initialization, and _env is not instantiated, so _env.stop() will not be executed, leading to the RpcServer not being closed.
Bug: SparkListenerBus not sending tableName details in logical plan for spark versions 3.4.2 and above
Status: Open | Priority: Critical | Resolution: 
Created: 22/May/24 13:00
Affects Version/s: 3.4.2

In OpenLineage, via SparkEventListener a logical plan event is received and by parsing it the frameworks deduces Input/Output table names to create a lineage.
The issue is that in spark versions 3.4.2 and above (tested and reproducible in 3.4.2 & 3.5.0) the logical plan event sent by spark core is partial and is missing the tableName property which was been sent in earlier versions (working in spark 3.3.4).


+_Note: This issue is only encountered in drop table events._+

For a drop table event, see below the logical plan in different spark versions

*Spark 3.3.4*
{code:java}
[
{
"class": "org.apache.spark.sql.execution.command.DropTableCommand",
"num-children": 0,
"tableName":

{ "product-class": "org.apache.spark.sql.catalyst.TableIdentifier", "table": "drop_table_test", "database": "default" }

,
"ifExists": false,
"isView": false,
"purge": false
}
]

{code}
*Spark 3.4.2*
{code:java}
[

{ "class": "org.apache.spark.sql.catalyst.plans.logical.DropTable", "num-children": 1, "child": 0, "ifExists": false, "purge": false }

,

{ "class": "org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier", "num-children": 0, "catalog": null, "identifier": null }

]

{code}
More details in referenced issue here: [https://github.com/OpenLineage/OpenLineage/issues/2716]
Bug: Spark stdout and stderr getting removed at end of spark job triggered from cloudera hue workflow
Status: Open | Priority: Critical | Resolution: 
Created: 09/May/24 10:04
Affects Version/s: 1.6.0

Hello,

we are running spark job triggered from cloudera hue workflow

and spark printing stdout and stderr logs during execution as expected:

e.g. -  !workflow_running_logs_printed.png!

 

*But stdout and stderr logs getting cleaned when workflows finished/status succeeded*

!workflow_succeeded_logs_cleaned.png!

 

following is spark-submit command workflow is triggering:

_/usr/bin/spark-submit _
  _--master yarn-client _
  _--driver-memory 4g _
  _--executor-memory 16g _
  _--executor-cores 4 _
  _--class tst _
  _--files `ls -m *.conf | tr -d '\n '` _
  _--conf "spark.dynamicAllocation.maxExecutors=4" _
  _--conf "spark.kryoserializer.buffer.max=1024" _
  _tst.jar $*_

 

 

does someone familiar with this spark job behavior or can advise ideas to fix it?

 

Thanks in advance
Bug: ErrorClassesJsonReader complies template regex on every template resolution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/24 12:33
Affects Version/s: 4.0.0

`SparkRuntimeException` uses `SparkThrowableHelper`, which uses `ErrorClassesJsonReader` to create error message string from templates in `error-conditions.json`, but template regex is compiled on every `SparkRuntimeException` constructor invocation.
Bug: Sql Scripting support for Spark SQL
Status: Reopened | Priority: Major | Resolution: 
Created: 20/May/24 09:51
Affects Version/s: 4.0.0

Design doc for this feature is in attachment.

High level example of Sql Script:
{code:java}
BEGIN
  DECLARE c INT = 10;
  WHILE c > 0 DO
    INSERT INTO tscript VALUES (c);
    SET c = c - 1;
  END WHILE;
END{code}
High level motivation behind this feature:
SQL Scripting gives customers the ability to develop complex ETL and analysis entirely in SQL. Until now, customers have had to write verbose SQL statements or combine SQL + Python to efficiently write business logic. Coming from another system, customers have to choose whether or not they want to migrate to pyspark. Some customers end up not using Spark because of this gap. SQL Scripting is a key milestone towards enabling SQL practitioners to write sophisticated queries, without the need to use pyspark. Further, SQL Scripting is a necessary step towards support for SQL Stored Procedures, and along with SQL Variables (released) and Temp Tables (in progress), will allow for more seamless data warehouse migrations.
Bug: Fix Flaky Test: `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
Status: Reopened | Priority: Major | Resolution: 
Created: 07/May/24 04:08
Affects Version/s: 4.0.0

This is a long standing flakiness from early 2024 to now.
- https://github.com/apache/spark/actions/runs/12882534288/job/35914995457 (2025-01-21)

{code}
- SPARK-43923: commands send events ((get_resources_command {
[info] }
[info] ,None)) *** FAILED *** (35 milliseconds)
[info]   VerifyEvents.this.listener.executeHolder.isDefined was false (SparkConnectServiceSuite.scala:873)
{code}
Bug: Fix UTF8String indexOf behaviour for empty string search
Status: Open | Priority: Major | Resolution: 
Created: 15/May/24 07:33
Affects Version/s: 4.0.0

Currently, UTF8String.indexOf returns 0 when given an empty parameters string, and any integer start value.

Examples:

{{"abc".indexOf("", 0);  // returns: 0}}

{{"abc".indexOf("", 2);  // returns: 0}}

{{"abc".indexOf("", 9);  // returns: 0}}

{{"abc".indexOf("", -3); // returns: 0}}

This is not correct, as "start" is not taken into consideration.

Correct behaviour would be:

{{"abc".indexOf("", 0);  // returns: 0}}

{{"abc".indexOf("", 2);  // returns: 2}}

{{"abc".indexOf("", 9);  // returns: -1}}

{{"abc".indexOf("", -3); // returns: -1}}
Bug: [M1] Exception handling
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/24 13:26
Affects Version/s: 4.0.0

Details TBD.

09.07.2024. - currently adding conditions and handlers as per the standard. try/catch syntax may be added later at some point

 

For more details:
 * Design doc in parent Jira item.
Bug: [Arbitrary State Support] Allow users to set EventTime timestamp properly in TransformWithState for watermark evaluation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/24 15:02
Affects Version/s: 4.0.0

TransformWithState operator allows users to provide a eventTimeColumn. This column is used for watermark calculation in any operation done after transformWithState.

Spark currently validates that the rows emitted from transformWithState operator adhere to the watermark boundary (specifically, watermarkForLateEvents). However, the user currently does not have a mechanism to obtain the watermarkForLateEvents.

We need to either 1) introduce a method to users which gives a watermark value before advancing (late events) or 2) provide a mechanism for users to set the event time timestamp properly without watermark value.
Bug: [M1] Support for ITERATE statement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/24 14:45
Affects Version/s: 4.0.0

Add support for ITERATE statement in WHILE (and other) loops to SQL scripting parser & interpreter.

This is the same functionality as CONTINUE in other languages.

 

For more details:
 * Design doc in parent Jira item.
Bug: [M0] Support for LEAVE statement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/24 12:58
Affects Version/s: 4.0.0

Add support for LEAVE statement in WHILE (and other) loops to SQL scripting parser & interpreter.

This is the same functionality as BREAK in other languages.

 

For more details:
 * Design doc in parent Jira item.
Bug: [M1] SQL Script execution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/24 12:36
Affects Version/s: 4.0.0

Implement changes to sql() API to support SQL script execution:
 * SparkSession changes
 * sql() API changes - iterate through the script, but return only last DataFrame
 * Spark Config flag to enable/disable SQL scripting in sql() API
 * E2E testing

-For more details, design doc can be found in parent Jira item.-

Have a verbal agreement with Wenchen on the design, need to create design-doc and POC PR.
Bug: Dynamic partition override mode might be ignored in certain scenarios causing data loss
Status: Open | Priority: Major | Resolution: 
Created: 29/May/24 10:09
Affects Version/s: 2.4.8

If an active spark session is stopped in the middle of an insert into file system, the session config responsible for overwriting partitions behavior might be not respected. The failure scenario basically is following:
 # The spark context is stopped just before  [getting a partition override mode setting|https://github.com/apache/spark/blob/8bbbde7cb3c396bc369c06853ed3a2ec021a2530/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala#L69]
 # Due to the [fallback config usage in case of stopped spark context,|https://github.com/apache/spark/blob/8bbbde7cb3c396bc369c06853ed3a2ec021a2530/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala#L121] this mode is evaluated to static (default mode in the default SQLConf used as  a fallback)
 # The data is cleared [here|https://github.com/apache/spark/blob/8bbbde7cb3c396bc369c06853ed3a2ec021a2530/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala#L131] totally which is literally a data loss from the user perspective who intends to overwrite data just partially.

This [gist|https://gist.github.com/akupchinskiy/b5f31781d59e5c0e9b172e7de40132cd] reproduces this behavior. On my local machine, it takes 1-3 iterations to have  pre-created data cleared totally.

The mitigation of this bug would be usage of explicit write parameter `partitionOverwriteMode`  instead of relying on session configuration.