av9ash/GitBugs huggingface dataset:
Bug: Spark Structured Streaming will process incorrect Kafka offsets
Status: Open | Priority: Critical | Resolution: 
Created: 04/Oct/24 15:01
Affects Version/s: 3.2.0

When using the Kafka structured streaming integration, KafkaOffsetReaderConsumer#fetchLatestOffsets checks if any of the offsets are incorrect, which is good behaviour. It has a set number of retries (which is configurable - `fetchoffset.numretries`), and a fetch retry interval (also configurable - `fetchoffset.retryintervalms`).

However, at the end of these retries, if the offsets are still incorrect, Spark uses the incorrect offsets to do the processing. For us this has caused processing to start at the beginning of the Kafka topic partition (as the incorrect offsets are usually 0), which is extremely undesirable and difficult to recover from (as the checkpointing then sets those values). 

The processing should stop after the set number of retries, as we do not want to be in a position that we start querying (possible years) old data. 

We are running version 3.2, but I can see that this logic is still in the master branch. In our current patched version, we have thrown an IllegalStateException if there are still incorrect offsets present in the incorrectOffsets collection. A la
{code:java}
if (incorrectOffsets.nonEmpty) {
          throw new IllegalStateException(s"Tried $attempt times (fetchoffset.numretries) to" +
            s" obtain valid offsets." +
            s" knownOffsets: $knownOffsets," +
            s" incorrectOffsets: $incorrectOffsets." +
            s" Aborting.")
} {code}
This change will, when the retries limit is exceeded and some offsets are still invalid, throw an IllegalStateException up through KafkaMicroBatchStream#latestOffset(), MicroBatchExecution#constructNextBatch(), and runActivatedStream() to StreamExecution#runStream() where it will be caught and terminate the query. 

There might be more elegant solutions available.
Bug: Unusual parameter behavior
Status: Open | Priority: Critical | Resolution: 
Created: 28/Oct/24 03:40
Affects Version/s: 3.5.3

dear, The parameters of spark-mission submitted by spark-submit  should not be parsed into commands but are parsed into commands，such as *spark.executor.extraJavaOptions，* *spark.executor.defaultJavaOptions,*  *spark.executor.extraJavaOptions,*  *spark.driver.defaultJavaOptions.* These parameters are used in the direct splice command and the command be executed in yarn mode (bash -c "xxxx" ).

*spark.executor.extraJavaOptions  is used in the direct splice command:*

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
{quote}private def prepareCommand(): List[String] = {
// Extra options for the JVM
val javaOpts = ListBuffer[String]()

// Set the JVM memory
val executorMemoryString = executorMemory + "m"
javaOpts += "-Xmx" + executorMemoryString

// Set extra Java options for the executor, if defined
sparkConf.get(EXECUTOR_JAVA_OPTIONS).foreach { opts =>
val subsOpt = Utils.substituteAppNExecIds(opts, appId, executorId)
javaOpts ++= Utils.splitCommandString(subsOpt).map(YarnSparkHadoopUtil.escapeForShell)
}
{quote}
 

From a security perspective, data and code (commands) should be handled separately. These parameters can be executed as codes (commands). A sugesstion for this issue that is recommended to verify these parameters to enhance security.
Bug: PySpark compatibility with Spark Connect
Status: Open | Priority: Critical | Resolution: 
Created: 01/Oct/24 23:12
Affects Version/s: 4.0.0

This aims to ensure full compatibility between PySpark and Spark Connect by thoroughly implementing and testing that all functionalities in PySpark work seamlessly with Spark Connect.

The [initial work|https://github.com/apache/spark/pull/48085] includes the creation of the *{{test_connect_compatibility.py}}* test suite, which validates the signature compatibility for core components such as {*}DataFrame{*}, {*}Column{*}, and *SparkSession* APIs. This test suite also includes checks for missing APIs and properties that need to be supported by Spark Connect.

Key goals for this project:
 * Ensure that all PySpark APIs are fully functional in Spark Connect.
 * Identify discrepancies in API signatures between PySpark and Spark Connect.
 * Verify missing APIs and properties, and add necessary functionality to Spark Connect.
 * Create comprehensive tests to prevent regressions and ensure long-term compatibility.

Further work will involve extending the test coverage to all critical PySpark modules and ensuring compatibility with Spark Connect in future releases.
Bug: Support end DeveloperAPI provided AgnosticEncoders based on ExpressionEncoders to allow migration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/24 16:17
Affects Version/s: 4.0.0

In Spark versions prior to 4.0.0-preview2 ExpressionEncoders were used throughout the original api.  SPARK-49025 removed re-use of ExpressionEncoders for derivation of tuples rendering user provided ExpressionEncoders such as Frameless incompatible with joinWith (which only uses the AgnosticEncoder hierarchy).  There are other likely combinations where the encoder is not passed into the function.

This patch aims to add an extension AgnosticExpressionPathEncoder trait with added:
{code:java}
def toCatalyst(input: Expression): Expression

def fromCatalyst(inputPath: Expression): Expression {code}
functions mirroring the underlying Des/SerializerBuildHelper approach (also used by Frameless).  In turn the BuildHelper's delegate to these functions when the AgnosticEncoder is an AgnosticExpressionPathEncoder.

Moreover it aims to fix AgnosticEncoder issues identified with moving Frameless to AgnosticEncoders instead of ExpressionEncoders allowing users to migrate via both the existing Expression codebase on 4.0 and future use of AgnosticEncoders (limited in the short term to Classic as serialization of derived codecs requires closures)
Bug: SPIP : Improving analyzer performance by skipping DeduplicateRelations rule conditionally
Status: Open | Priority: Major | Resolution: 
Created: 04/Oct/24 18:41
Affects Version/s: 3.5.3

In many cases, it has been observed that DeduplicateRelations rule, though essential, but by its nature has impacted query analysis time to big extent especially when dealing with large query plans.

It appears that in many situations we can guarantee that there would be no duplicate relations present and thus avoid applying the rule.

Those situations are :

1) When dataframe api's like select/filter which operate on existing dataframe, are used.

 

Also if we store the MultiInstanceRelations in the QueryExecution ,for  a given plan, then we can make use of that information , while creating new dataframes where there is a possibility of duplicate relations ( like join, union, intersection etc).

If two datasets being unioned/intersected/joined ..etc have no common MultiInstanceRelation , then it should be safe to assume that there is no possibility of duplicate relations , there by allowing skipping the Dedup rule

 

Atleast that is the idea.
Bug: KafkaWriteTask: support timestamp customisation 
Status: Open | Priority: Major | Resolution: 
Created: 29/Oct/24 14:21
Affects Version/s: 3.5.3

Currently, there is no way to customise the timestamp of a {{ProducerRecord}} produced by the {{{}KafkaWriteTask{}}}. Here at Wikimedia we often use event-time semantics, so it would be helpful if Kafka records produced via spark could resemble that.

The producer already allows that as stated in the [{{Producer.send}}|#send-org.apache.kafka.clients.producer.ProducerRecord-org.apache.kafka.clients.producer.Callback-] API docs:
{quote}If CreateTime is used by the topic, the timestamp will be the user provided timestamp or the record send time if the user did not specify a timestamp for the record.
{quote}
So the proposed feature enables users of the spark kafka output to specify the create-time.

I already checked out the code and was able to adapted it to fulfil our need. Since I couldn't find any (closed) ticket concerned with this topic, I assume no such feature has been denied until now.
Bug: PySpark expr() (expression) SQL Function returns None in Spark Connect
Status: Open | Priority: Major | Resolution: 
Created: 25/Oct/24 17:11
Affects Version/s: 3.5.0

When PySpark is launched with remote Spark Connect configuration, calls to pyspark.sql.functions.expr always return `Column<None>`.

 

PySpark without Spark Connect
{code:java}
$ pyspark
...
>>> from pyspark.sql.functions import expr
>>> e = expr("*")
>>> print(e)
Column<'unresolvedstar()'>
{code}
PySpark with Spark Connect
{code:java}
$ pyspark --remote "sc://localhost:15002"
...
Client connected to the Spark Connect server at localhost:15002
SparkSession available as 'spark'.
>>> from pyspark.sql.functions import expr
>>> e = expr("*")
>>> print(e)
Column<'None'>
{code}
Additional example
{code:java}
$ pyspark
...
Spark context available as 'sc' (master = local[*], app id = local-1729876117745).
SparkSession available as 'spark'.
>>> from pyspark.sql.functions import expr
>>> expr("*")
Column<'unresolvedstar()'>
>>> SparkSession.builder.master("local[*]").getOrCreate().stop()
>>> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
>>> expr("*")
Column<'None'> {code}
Bug: Avoid fallback to Hive-incompatible ways when table creation fails by thrift exception
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/24 16:15
Affects Version/s: 3.5.3

When creating datasource table by {{HiveExternalCatalog}}, Spark would fall back to create table in hive incompatible way for all nonfatal exceptions. This is incorrect for thrift exceptions caused by connection issues, where we could still attempt to create table in hive  compatible way by retry mechanism.
Bug: The split function produces incorrect results with an empty regex and a limit
Status: Open | Priority: Major | Resolution: 
Created: 15/Oct/24 04:27
Affects Version/s: 3.5.1

The current behavior of the split function is as follows:
{code:java}
select split('hello', 'h', 1) // result is ["hello"]
select split('hello', '-', 1) // result is ["hello"]
select split('hello', '', 1)  // result is ["h"]

select split('1A2A3A4', 'A', 3) // result is ["1","2","3A4"]
select split('1A2A3A4', '', 3)  // result is ["1","A","2"]{code}
However, according to the function's description, when the limit is greater than zero, the last element of the split result should contain the remaining part of the input string.
{code:java}
Arguments:
      * str - a string expression to split.
      * regex - a string representing a regular expression. The regex string should be a Java regular expression.
      * limit - an integer expression which controls the number of times the regex is applied.
          * limit > 0: The resulting array's length will not be more than `limit`, and the resulting array's last entry will contain all input beyond the last matched regex.
          * limit <= 0: `regex` will be applied as many times as possible, and the resulting array can be of any size. {code}
So, the split function produces incorrect results with an empty regex and a limit. The correct result should be: 
{code:java}
select split('hello', '', 1)    // result is ["hello"]

select split('1A2A3A4', '', 3)  // result is ["1","A","2A3A4"]{code}
Bug: Rename errorClass to condition in JdbcDialect.classifyException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/24 21:37
Affects Version/s: 4.0.0

Rename "errorClass" to "condition" in JdbcDialect.classifyException to follow up the agreement of https://issues.apache.org/jira/browse/SPARK-46810
Bug: Build should be stopped when Antrun script fails
Status: Open | Priority: Major | Resolution: 
Created: 20/Oct/24 10:48
Affects Version/s: 3.5.3

When I compiled the spark-core module in the windows environment, the bash/powershell program could not be executed because my operating system had WSL installed, and the build information file was not generated in the spark-core jar. When I used this jar for verification, I found that the jar was not working properly as the build information was not present.

Therefore, I think when the antrun script in spark-core module fails, the build progress should be ended immediately, otherwise the builded `spark-core.jar` will not work properly.
Bug: Spark supports executor HostNetwork on k8s mode
Status: Open | Priority: Major | Resolution: 
Created: 20/Oct/24 14:29
Affects Version/s: 4.0.0

In k8s mode, when the executor executes the host network mode, it means that the executor shares the same IP address with the host. To support this mode in Spark, spark need to allow the Executor to be launched using a random BlockManager port. And prohibit shuffle local read, as this will cause the shuffle to be read failed.
Bug: Dataset.transform no longer has the correct return type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/24 16:53
Affects Version/s: 4.0.0

In versions prior to 4.0.0-preview2 sql.Dataset transform had signature:
{code:java}
def transform[U](t: (sql.Dataset[T]) ⇒ sql.Dataset[U]): sql.Dataset[U] {code}
4.0.0-preview2 has moved this to the parent class sql.api.Dataset with the signature:
{code:java}
def transform[U](t: (sql.api.Dataset[T]) ⇒ sql.api.Dataset[U]): sql.api.Dataset[U] {code}
rendering all function objects and return values with incompatible types.

It seems F Bounded or some similar self type is needed to have the types remain correct (e.g. if you are dealing with sql.Dataset all types should be sql.Dataset),
{code:java}
import sparkSession.implicits._
val ds = Seq(1, 2).toDS()
val f: Dataset[Int] => Dataset[Int] = d => d.selectExpr("(value + 1) value").as[Int]
val transformed = ds.transform(f)
assert(transformed.collect().sorted === Array(2, 3)) {code}
now fails to compile.
Bug: Query fails when aggregate expression is in left-hand operand of IN-subquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/24 16:52
Affects Version/s: 3.5.2

Consider this query:
{noformat}
create or replace temp view v1(c1, c2) as values
(1, 2), (1, 3), (2, 2), (3, 7), (3, 1);
create or replace temp view v2(col1, col2) as values
(1, 2), (1, 3), (2, 2), (3, 7), (3, 1);

select col1, sum(col2) in (select c2 from v1)
from v2 group by col1;
{noformat}
It fails with the following error:
{noformat}
[INTERNAL_ERROR] Cannot generate code for expression: sum(input[1, int, false]) SQLSTATE: XX000
{noformat}
With SPARK_TESTING=1, it fails with
{noformat}
[PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule org.apache.spark.sql.catalyst.optimizer.RewritePredicateSubquery in batch RewriteSubquery generated an invalid plan: Special expressions are placed in the wrong plan: Aggregate [col1#11], [col1#11, first(exists#20, false) AS (sum(col2) IN (listquery()))#19]
+- Join ExistenceJoin(exists#20), (sum(col2#12) = c2#18L)
   :- LocalRelation [col1#11, col2#12]
   +- LocalRelation [c2#18L]
{noformat}
The {{Join}} is invalid because it contains an aggregate expression in the join condition.

The bug is in the handler for UnaryNode in {{RewritePredicateSubquery#apply,}} which adds a {{Join}} below the {{Aggregate}} and assumes that the left-hand operand of IN-subquery can be used in the join condition. This works fine for most cases, but not when the left-hand operand is an aggregate expression.
Bug: make "limit n" in spark connect use executeCollect
Status: Open | Priority: Major | Resolution: 
Created: 12/Oct/24 09:08
Affects Version/s: 3.5.3

```SQL

spark.sql("select * from some_tables limit 10").collect()

```

In spark connect, this sql will call `SparkPlan.execute` and run all partitions. But in spark it will call `executeCollect` which is more performant for `CollectLimitExec` and `CollectTailExec`
Bug: Spark removes working directory while Python UDF runs
Status: Reopened | Priority: Major | Resolution: 
Created: 25/Oct/24 13:17
Affects Version/s: 3.5.2

With Spark Connect + PySpark, we can stage files using `spark.addArtifacts`. When a Python UDF is executed, the working directory is set to a folder with the corresponding artifacts available.

I have observed on large scale jobs with long running tasks (>45 mins) that Spark sometimes removes that working directory, even though UDF tasks are still running. This can be seen by periodically running `os.getcwd()` in the UDF, which raises `FileNotFoundError`.

This seems to coincide with log records indicating 'Session evicted: <uuid>`, from [`isolatedSessionCache`.|#L212] There is a 30 minute timeout here that might be to blame.

I have not yet been able to write a simple program to reproduce. I suspect that there might be a conjunction of multiple events, such as when a task is scheduled on an executor 30 mins after the last task started. https://issues.apache.org/jira/browse/SPARK-44290 might be relevant.

cc [~gurwls223]
Bug: Initial PR to support spark.ml on Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/24 09:21
Affects Version/s: 4.0.0

Starting from Apache Spark 3.4, Spark has supported Connect which introduced a decoupled client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. The separation between client and server allows Spark and its open ecosystem to be leveraged from everywhere. It can be embedded in modern data applications, in IDEs, Notebooks and programming languages.

However, Spark Connect currently only supports Spark SQL, which means Spark ML could not run the training/inference via Spark Connect. It will probably result in losing some ML users.

So I would like to propose a way to support Spark ML on the Connect. Users don't need to change their code to leverage connect to run Spark ML cases.

Here are some links,

Design doc: [Support spark.ml on Connect|https://docs.google.com/document/d/1EUvSZuI-so83cxb_fTVMoz0vUfAaFmqXt39yoHI-D9I/edit?usp=sharing] 

Draft PR: https://github.com/wbo4958/spark/pull/5

Example code,


{code:python}
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

df = spark.createDataFrame([
    (Vectors.dense([1.0, 2.0]), 1), 
    (Vectors.dense([2.0, -1.0]), 1), 
    (Vectors.dense([-3.0, -2.0]), 0), 
    (Vectors.dense([-1.0, -2.0]), 0), 
], schema=['features', 'label'])

lr = LogisticRegression()
lr.setMaxIter(30)

model: LogisticRegressionModel = lr.fit(df)
z = model.summary
x = model.predictRaw(Vectors.dense([1.0, 2.0]))
print(f"predictRaw {x}")
assert model.getMaxIter() == 30
model.summary.roc.show()

print(model.summary.weightedRecall)
print(model.summary.recallByLabel)
print(model.coefficients)
print(model.intercept)

model.transform(df).show()
{code}
Bug: SpecialLimits strategy doesn't work when return the content of the Dataset as a Dataset of JSON strings
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/24 06:53
Affects Version/s: 3.5.3

`CollectLimitExec` is used when a logical `Limit` and/or `Offset` operation is the final operator. Comparing to `GlobalLimitExec`, it can avoid shuffle data to a single output partition.
But when the dataset is collected as a Dataset of JSON strings. The `GlobalLimitExec` and `TakeOrderedAndProjectExec` are not able to applied since the SpecialLimits strategy cannot work as expected.
Here is an example:

{code:java}
scala> spark.sql("select * from right_t limit 4").explain
== Physical Plan ==
CollectLimit 4
+- Scan hive spark_catalog.default.right_t [id#23, name#24], HiveTableRelation [`spark_catalog`.`default`.`right_t`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#23, name#24], Partition Cols: []]

scala> spark.sql("select * from right_t limit 4").toJSON.explain
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true, false, true) AS value#34]
   +- MapPartitions org.apache.spark.sql.Dataset$$Lambda/0x00000070021d8c58@5b17838a, obj#33: java.lang.String
      +- DeserializeToObject createexternalrow(staticinvoke(class java.lang.Integer, ObjectType(class java.lang.Integer), valueOf, id#27, true, false, true), name#28.toString, StructField(id,IntegerType,true), StructField(name,StringType,true)), obj#32: org.apache.spark.sql.Row
         +- GlobalLimit 4, 0
            +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=42]
               +- LocalLimit 4
                  +- Scan hive spark_catalog.default.right_t [id#27, name#28], HiveTableRelation [`spark_catalog`.`default`.`right_t`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#27, name#28], Partition Cols: []]
{code}
Bug: ExecutorFailureTracker sometimes miss failed executors on k8s
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/24 23:31
Affects Version/s: 3.5.3

On k8s currently the executor failure tracking is done in the ExecutorPodsAllocator class which calculates with the failed PODs (where POD state is PodFailed) but ExecutorPodsLifecycleManager can remove those pods when "spark.kubernetes.executor.deleteOnTermination" is switched on.

This can reproduced easily by making the executor fail and check the executor IDs known by ExecutorFailureTracker by adding some new logs:

{noformat}
24-09-25 20:29:51 WARN ExecutorPodsAllocator: all failed: Set(69, 5, 10, 56, 37, 52, 20, 46, 78, 74, 70, 21, 33, 53, 77, 73, 32, 34, 45, 22, 71, 54, 49, 76, 91, 66, 35, 48, 18, 50, 67, 11, 72, 55, 75, 36, 51, 19, 47, 68, 90)
24-09-25 20:29:51 ERROR ExecutorPodsAllocator: Max number of executor failures (40) reached       

{noformat}

You can see 40 was the limit but the "all failed" set does not contain the first 40 consecutive executor IDs but there are huge gaps (even executor ID 91 is contained).
Bug: spark TLS connection (+ kerberos) to hive metastore
Status: Open | Priority: Major | Resolution: 
Created: 09/Oct/24 08:09
Affects Version/s: 3.5.2

Hi,
we are trying to configure an integration between trino, spark and hive metastore (HMS) in a secure way.
 
Hive metastore has already been configured in order to use kerberos and TLS.
Trino has already been configured in order to connect to HMS using TLS and kerberos.
 
Trying to do the same for spark (connect it to HMS using TLS and kerberos) we faced a problem with TLS connection: if we configure spark using kerberos and plain connection to HMS (reconfiguring HMS too) it works, but if we enable TLS on both, spark is not able to connect.
 
The error on HMS is the following: "Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message" and indeed connections initiated by spark are alway plain.
 
The test matrix is the following:
 # hive (kerberos + ssl) + spark (kerberos + ssl) --> not working
 # hive (kerberos + plain) --> spark (kerberos + plain) --> works
 # hive (ssl) ---> spark (ssl) --> works

 
While doing "test 1", I also used tcpdump to figure out if spark was trying to start an ssl or a plain connection to hive, and for what I'm seeing spark is completely ignoring the following parameters and keep trying to open a plain connection:
{code:java}
spark.hive.metastore.use.SSL true
spark.hive.metastore.truststore.path /opt/spark/ssl/cert.jks
spark.hive.metastore.truststore.password mypassword{code}
If I enable both kerberos and ssl (test 1), it seems like those hive ssl related configurations on spark-defaults are being ignored and spark always tries to open a plain connection; for example, If I set "spark.hive.metastore.truststore.password" to "wrongpassword" the error "Password verification failed" should be raised, but nothing
 
spark conf: [https://gist.github.com/bovy89/83cbe3b9cd7a318fa9fd35355d5801fc]
pyspark logs: [https://gist.github.com/bovy89/a06f0aa4a54f454fea9e0d6ff148cfc5#file-pyspark-log]
pyspark debug logs: [https://gist.github.com/bovy89/e6a9eeca389f05ff7bea78f807ce5714]
hive metastore logs: [https://gist.github.com/bovy89/a06f0aa4a54f454fea9e0d6ff148cfc5#file-hive-metastore-log]