av9ash/GitBugs huggingface dataset:
Bug: Remove the ANSI config suggestion in NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/24 10:19
Affects Version/s: 4.0.0

NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try_*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: Remove the ANSI config suggestion in CAST_INVALID_INPUT and CAST_OVERFLOW
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/24 06:40
Affects Version/s: 4.0.0

CAST_INVALID_INPUT and CAST_OVERFLOW both contain suggested fixes for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exists implementation of `try_cast` which provides casting as for ANSI mode off and that suggestion should be sufficient for users to move forward.
Bug: Spark reading data in scientific notation in String field
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/24 13:53
Affects Version/s: 3.1.0

When I'm reading some json payloads PySpark is changing the data even if I read it as a StringType and I want this as a String because I don't want to have each field as a column at this step. I just want to get this payload as String as it is in payload/source file
Locally I'm using Spark 3.3 in Jupiter Notebook with Glue 4 image PySpark version: 3.3.0+amzn.1.dev0

Here my payload/source (test.txt):
{code:java}
{"payload":{"points":1220000000}}
{"payload":{"count":1550554545.0}}
{"payload":{"points":125888002540.0, "count":1550554545.0}}
{"payload":{"name": "Roger", "count":55154111.0}}{code}
Here my code:
{code:java}
path = "/home/glue_user/workspace/jupyter_workspace/test/test.txt"
schema = StructType([StructField('payload', StringType(), True)])
my_df = spark.read.schema(schema).option("inferSchema", "false").json(path)
my_df.show(truncate=False){code}
Here the result where PySpark is setting the float number in scientific notation, even when I read it as String.
{code:java}
+------------------------------------------------+
|payload                                         |
+------------------------------------------------+
|{"points":1220000000}                           |
|{"count":1.550554545E9}                         |
|{"points":1.2588800254E11,"count":1.550554545E9}|
|{"name":"Roger","count":5.5154111E7}            |
+------------------------------------------------+ {code}
Why I can't simply have my data as it is? Why the final result is changed into my string field and receive this scientific notation? i.e:
{quote}"count":1550554545.0
"count":1.550554545E9
{quote}
Bug: Feature transformers are case sensitive
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/24 12:29
Affects Version/s: 3.5.2

Hi team,

[https://spark.apache.org/docs/latest/ml-features]

The feature transformers are case sensitive even though the configuration 
{code:java}
spark.conf.get("spark.sql.caseSensitive") {code}
is set to false. The user of all these transformers are forced to abide by case of the column in the dataframe
{code:java}
  val data = List(Row("the movie was great", "positive", 10, "greatest of all time"),
    Row("the movie was average", "negative", 11, "just average things, average storyline"),
    Row("movie was fun", "positive", 2, "superb screen play"))
  val schema = new StructType()
    .add("comments", StringType, true)
    .add("reviews", StringType, true)
    .add("counts", IntegerType, true)
    .add("Additional_COMMENTS", StringType, true)
val df = spark.createDataFrame(data.asJava, schema)
  val si = new StringIndexer().setInputCol("additional_comments").setOutputCol("si_additional_comments")
  si.fit(df).transform(df).show() {code}
The above code fails with 
{code:java}
Exception in thread "main" org.apache.spark.SparkException: Input column additional_comments does not exist.
    at org.apache.spark.ml.feature.StringIndexerBase.$anonfun$validateAndTransformSchema$2(StringIndexer.scala:128)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
    at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
    at org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema(StringIndexer.scala:123)
    at org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema$(StringIndexer.scala:115) {code}
Which means that the column "additional_comments" needs to be provided in the same case as in the dataframe. 



I think when the caseSensitive  setting is set to *false* we should be able to use the naming in any case.

 

Can someone please help to solve this bug for all transformers.?
Bug: Feature transformers are case sensitive
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/24 12:28
Affects Version/s: 3.5.2

Hi team,

[https://spark.apache.org/docs/latest/ml-features]

The feature transformers are case sensitive even though the configuration 
{code:java}
spark.conf.get("spark.sql.caseSensitive") {code}
is set to false. The user of all these transformers are forced to abide by case of the column in the dataframe
{code:java}
  val data = List(Row("the movie was great", "positive", 10, "greatest of all time"),
    Row("the movie was average", "negative", 11, "just average things, average storyline"),
    Row("movie was fun", "positive", 2, "superb screen play"))
  val schema = new StructType()
    .add("comments", StringType, true)
    .add("reviews", StringType, true)
    .add("counts", IntegerType, true)
    .add("Additional_COMMENTS", StringType, true)
val df = spark.createDataFrame(data.asJava, schema)
  val si = new StringIndexer().setInputCol("additional_comments").setOutputCol("si_additional_comments")
  si.fit(df).transform(df).show() {code}
The above code fails with 
{code:java}
Exception in thread "main" org.apache.spark.SparkException: Input column additional_comments does not exist.
    at org.apache.spark.ml.feature.StringIndexerBase.$anonfun$validateAndTransformSchema$2(StringIndexer.scala:128)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
    at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
    at org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema(StringIndexer.scala:123)
    at org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema$(StringIndexer.scala:115) {code}
Which means that the column "additional_comments" needs to be provided in the same case as in the dataframe. 



I think when the caseSensitive  setting is set to *false* we should be able to use the naming in any case.

 

Can someone please help to solve this bug for all transformers.?
Bug: Basic admission control over the number of concurrent executions
Status: Open | Priority: Major | Resolution: 
Created: 10/Sep/24 07:31
Affects Version/s: 4.0.0

Related to https://issues.apache.org/jira/browse/SPARK-49544

Need to limit the number of ExecuteHolders associated to a single session.
-> If the limit is reached, the client should retry (add a new error class + modify RetryPolicy.scala).

*I'd like to emphasise that this can be a (D)DoS attack vector! A single session can spawn unlimited number of threads!*.

+ (Maybe) The number of total sessions should also be limited.
Bug: OperatorStateMetadataV2 can return incorrect metadata for the given batchId
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 10/Sep/24 21:28
Affects Version/s: 4.0.0

Currently, the `OperatorStateMetadataV2Reader` has an off-by-one error when reading metadata for a given batchId. The issue is on [this line|https://github.com/apache/spark/blob/ea4b82e97d944717b32aaaec986a8238e59b8acf/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/OperatorStateMetadata.scala#L349].

Suppose the batches in the checkpoint location is just `0`, and the desired `batchId` is 1. The logic lists all of the batches, filters for less than `batchId`, and takes the last element. This can return the metadata for `0`, when really we want the metadata for `1`. The reason we didn't catch this before was because [this check|https://github.com/apache/spark/blob/ea4b82e97d944717b32aaaec986a8238e59b8acf/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/OperatorStateMetadata.scala#L316-L318] prevented us from getting to the point where we attempted to read metadata for a batch whose offset didn't yet exist.
Bug: Severe lock contention in SparkConnectExecutionManager
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/24 09:37
Affects Version/s: 4.0.0

Critical sections protected by executionsLock can become too broad when there are too many ExecuteHolders, e.g., >= 10^4. The problem is aggravated when there are too many threads in the system: priority inversion. In order to minimise the chance of a thread getting pre-empted holding executionsLock, replace it with a concurrent hash map that internally partitions the data: coarse-grained locking -> fine-grained locking.

-> https://issues.apache.org/jira/browse/SPARK-49580 will be eventually needed.
Bug: Merge options of table and relation when create FileScanBuilder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/24 02:47
Affects Version/s: 4.0.0

Currently, the subclass of {{FileTable}} only uses the options from relation when constructing the {{{}FileScanBuilder{}}}, which leads to the omission of the contents in {{{}FileTable.options{}}}. For the {{{}TableCatalog{}}}, the {{dsOptions}} can be set into the {{FileTable.options}} returned by the {{TableCatalog.loadTable}} method. If only the relation options are used here, the {{TableCatalog}} will not be able to pass {{dsOptions}} that contains table options to {{{}FileScan{}}}.

Merge the two options is a better option.
Bug: Improve UDF Packet Serialization
Status: Open | Priority: Major | Resolution: 
Created: 10/Sep/24 16:30
Affects Version/s: 4.0.0

When looking at serialized UDF packets it is noticeable that agnostic encoders are taking up quite a bit of space. In many cases they capture quite a bit more than expected (a lot of the objects capture things in their class hierarchy.

Since we need to be backwards compatible here, we can't just change serialization. Instead the proposal would be to use Serialization proxies for UdfPacket, and perhaps the encoders themselves.
Bug: datetime formatter violates assert of variable seconds fraction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/24 08:17
Affects Version/s: 4.0.0

The datetime pattern "\nSSSS\r" cause failure on assert while parsing the pattern by datetime formatter:

{code}
java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:264)
	at org.apache.spark.ErrorClassesJsonReader.getMessageTemplate(ErrorClassesJSONReader.scala:91)
	at org.apache.spark.ErrorClassesJsonReader.getErrorMessage(ErrorClassesJSONReader.scala:46)
	at org.apache.spark.SparkThrowableHelper$.getMessage(SparkThrowableHelper.scala:56)
	at org.apache.spark.SparkIllegalArgumentException.<init>(SparkException.scala:405)
	at org.apache.spark.SparkIllegalArgumentException.<init>(SparkException.scala:417)
	at org.apache.spark.SparkIllegalArgumentException.<init>(SparkException.scala:422)
	at org.apache.spark.sql.catalyst.util.DateTimeFormatterHelper$.$anonfun$createBuilderWithVarLengthSecondFraction$1(DateTimeFormatterHelper.scala:247)
{code}
Bug: Generated code throws Exception without line number
Status: Open | Priority: Major | Resolution: 
Created: 10/Sep/24 07:34
Affects Version/s: 3.2.1

There is no line number information for Exceptions thrown in generated code.

Only shows (Unknown Source).
{code:java}
// code placeholder
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_2_5$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.FileSourceScanExec.$anonfun$doExecute$4(DataSourceScanExec.scala:700)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:180)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:61)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:134)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:538)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1618)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:541) {code}
Bug: Upload Python logs in CI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/24 04:06
Affects Version/s: 4.0.0

e.g., 

/__w/spark/spark/python/target/28a23950-46c7-45c5-a9b7-42e7d9b21518/python3.12__pyspark.sql.tests.connect.test_connect_session__ah_ug0xu.log)
Bug: Add error handling for python streaming data source record prefetching
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/24 07:01
Affects Version/s: 4.0.0

Currently there is an assert that return status code from the python worker is SpecialLengths.START_ARROW_STREAM when python source runner is prefetching records. To improve debugability, check the status code and rethrow an runtime error with detailed error message.
Bug: Revamp Connect MiMa tests and make it work with the new interface.
Status: Open | Priority: Major | Resolution: 
Created: 10/Sep/24 00:35
Affects Version/s: 4.0.0

The current connect MiMa tests are the only way we can guarantee that the connect client and the classic client expose the same API. The shared interface will take care of most of this now. However we should still have checks running to make sure the implementation are not exposing things that the interface or the other implementation is missing.
Bug: Catalog createTable API is double-escaping paths
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/24 14:17
Affects Version/s: 4.0.0

Creating an external table using {{spark.catalog.createTable}} results in incorrect escaping of special chars in paths.

Consider the following code:

{{{{{}spark.catalog.createTable({}}}{{{}"testTable", source = "parquet", schema = new StructType().add("id", "int"), description = "", options = Map("path" -> "/tmp/test table")){}}}}}

The above call creates a table that is stored in {{/tmp/test%20table}} instead of {{{}/tmp/test table{}}}. Note that this behaviour is different from the SQL API, e.g. {{create table testTable(id int) using parquet location '/tmp/test table'}}
Bug: Improve documentation of "plotting.sample_ratio" option
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/24 07:09
Affects Version/s: 4.0.0

The current documentation incorrectly suggests that "plotting.sample_ratio" defaults to "plotting.max_rows". In reality, if "plotting.sample_ratio" is not explicitly set, it is *derived* based on the ratio of "plotting.max_rows" to the dataset size.
Bug: Hang: reattach attempted without ExecuteHolder started after exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/24 12:00
Affects Version/s: 4.0.0

# ExecuteHolder.eventManager.postStarted throws an exception, e.g., OoM.
 # Reattach attempted.
 # Reattach creates an ExecuteGrpcResponseSender, which will never be responded.
 # Hang.
Bug: Windows-style paths are unsupported in ArtifactManager
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/24 21:11
Affects Version/s: 4.0.0

Currently, windows-based clients will run into an issue when using the `addArtifact` API as the path passed to the server would contain backslashes which the server would interpret as part of the file name rather than a separator. 

E.g if the client sends the name `pyfiles\abc.txt` to the server, then the artifact would be written out as `<aritfact_directory>/pyfiles\abc.txt` instead of the correct `<artifact_directory>\pyfiles\abc.txt
Bug: ProtoToParsedPlanTestSuite should recognize rules in BaseSessionStateBuilder.scala
Status: Open | Priority: Major | Resolution: 
Created: 02/Sep/24 02:39
Affects Version/s: 4.0.0

A new Analyzer is created in the test suite ProtoToParsedPlanTestSuite, see [here|https://github.com/apache/spark/blob/master/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala#L160].

This is why it didn’t recognize the rule added in BaseSessionStateBuilder, for example, [here|https://github.com/apache/spark/pull/47884/files#diff-9806431743675ca892eb73a801af2f4c43086f87ecbc0c94900c8f18660f4a50R208].

Overriding extendedResolutionRules in the new Analyzer of the test suite is not feasible cause the SparkSession is null at the moment, leading to

 

{{[info]   java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.SparkSession.sessionState()" because "this.$outer.org$apache$spark$sql$catalyst$analysis$ResolveTranspose$$sparkSession" is null}}

 

[https://github.com/apache/spark/pull/47884/files#diff-18773e9500b5f13ce74b6f9c01bfee44b6b5a70fc3378997cfb691c503d87bdaR182] should be enabled after the fix of test suite.
Bug: Spark Streaming Exposes Security Kafka Keys to Logs
Status: Open | Priority: Major | Resolution: 
Created: 03/Sep/24 09:10
Affects Version/s: 3.5.1

 

I have a Spark Streaming job that reads data from a Delta table, performs transformations, and writes the data to Kafka in Avro format. However, in the event of a failure, Spark logs the logical plan, which inadvertently includes all Kafka properties, including sensitive mTLS keys.

The code:
{code:java}
val df = spark.readStream.format("delta")
      .option("ignoreCorruptFiles", "true")
      .option("path", fullPath)
      .schema(schema.get)
      .load()
val transformedDF = df.transform(...)  
 transformedDF.writeStream
      .format("kafka")
      .option("checkpointLocation", cpPath)
      .options(config.conf)
      .trigger(trigger.getOrElse(Trigger.ProcessingTime(0L)))
      .start() 
 df.sparkSession.streams.awaitAnyTermination(){code}
The exception:
{code:java}
Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException: org.apache.hadoop.fs.s3a.RemoteFileChangedException: open s3a://.../_delta_log/00000000000001258632.json at 0 on s3a://.../_delta_log/00000000000001258632.json: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 412, Request ID: tx0000012a53fe47376ba53-0066cf6372-149bb8-default):PreconditionFailed `s3a://.../_delta_log/00000000000001258632.json': : null (Service: S3, Status Code: 412, Request ID: tx0000012a53fe47376ba53-0066cf6372-149bb8-default) === Streaming Query === Identifier: [id = 0d9cd948-f417-4688-8c02-355a772299a4, runId = f420b516-3e25-4443-8ae3-9f511b37ff8c] Current Committed Offsets: {DeltaSource[s3a://...]: {"sourceVersion":1,"reservoirId":"4c9d94b9-37b7-4aa7-bc9e-e914dfdc4ea2","reservoirVersion":1258632,"index":-1,"isStartingVersion":false}} Current Available Offsets: {DeltaSource[s3a://...]: {"sourceVersion":1,"reservoirId":"4c9d94b9-37b7-4aa7-bc9e-e914dfdc4ea2","reservoirVersion":1258632,"index":-1,"isStartingVersion":false}}Current State: ACTIVE Thread State: RUNNABLE{code}
{code:java}
Logical Plan: WriteToMicroBatchDataSource org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaTable@7945310f, 0d9cd948-f417-4688-8c02-355a772299a4, [kafka.ssl.keystore.type=PEM, subscribe=dns-correlated-events, kafka.ssl.truststore.certificates=-----BEGIN CERTIFICATE----- XXXX -----END CERTIFICATE----- , kafka.max.request.size=2000000, kafka.bootstrap.servers=kafkacluster-kafka-bootstrap.kafka.svc.cluster.local:9094, kafka.ssl.keystore.key=-----BEGIN PRIVATE KEY----- XXXX -----END PRIVATE KEY----- , checkpointLocation=s3a://…, topic=dns-correlated-events, kafka.ssl.truststore.type=PEM, kafka.ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- XXXX -----END CERTIFICATE----- , kafka.security.protocol=SSL], Append +- Project [to_aero(struct(ts, ts#744L, ...
{code}
Bug: [CSV Reader] Add Flag to Control Inference of Time-Only Columns as String or Timestamp During Schema Detection
Status: Open | Priority: Minor | Resolution: 
Created: 24/Sep/24 03:58
Affects Version/s: 3.5.2

This task involves adding a configurable flag to control the inference of time-only columns during schema detection in Spark. By default, Spark converts time-only columns to {{Timestamp}} type, which can lead to unintended behavior in certain use cases. This new flag will allow users to specify whether time-only columns should be inferred as {{Timestamp}} or as {{{}String{}}}.
h3. Key Changes:
 * Introduce a flag (e.g., {{{}inferStringTypeForTimeOnlyColumn{}}}).
 * When the flag is set to {{{}true{}}}, time-only columns will be inferred as {{{}String{}}}.
 * When the flag is set to {{false}} (default), time-only columns will be inferred as {{{}Timestamp{}}}.
 * Update documentation to reflect the new option.
 * Ensure backward compatibility by defaulting to the current behavior.
Bug: Assign a name to the error conditions _LEGACY_ERROR_TEMP_3055, 3146
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Sep/24 15:38
Affects Version/s: 4.0.0

Choose a proper name for the error conditions *_LEGACY_ERROR_TEMP_3055* and *_LEGACY_ERROR_TEMP_3146* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-conditions.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-conditions.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]