av9ash/GitBugs huggingface dataset:
Bug: Trigger `committer.setupJob` before plan execute in `FileFormatWriter`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/23 07:11
Affects Version/s: 3.2.3

In this jira, the case where `outputOrdering` might not work if AQE is enabled has been resolved.

https://issues.apache.org/jira/browse/SPARK-40588

However, since it materializes the AQE plan in advance (triggers getFinalPhysicalPlan) , it may cause the committer.setupJob(job) to not execute When `AdaptiveSparkPlanExec#getFinalPhysicalPlan()` is executed with an error.

Normally this step should be executed after committer.setupJob(job).

This may eventually result in the insertoverwrite directory being deleted.

 
{code:java}
import org.apache.hadoop.fs.{FileSystem, Path}

import org.apache.spark.sql.QueryTest
import org.apache.spark.sql.catalyst.TableIdentifier

sql("CREATE TABLE IF NOT EXISTS spark32_overwrite(amt1 int) STORED AS ORC")
sql("CREATE TABLE IF NOT EXISTS spark32_overwrite2(amt1 long) STORED AS ORC")
sql("INSERT OVERWRITE TABLE spark32_overwrite2 select 6000044164")
sql("set spark.sql.ansi.enabled=true")

val loc =
  spark.sessionState.catalog.getTableMetadata(TableIdentifier("spark32_overwrite")).location

val fs = FileSystem.get(loc, spark.sparkContext.hadoopConfiguration)
println("Location exists: " + fs.exists(new Path(loc)))

try {
  sql("INSERT OVERWRITE TABLE spark32_overwrite select amt1 from " +
    "(select cast(amt1 as int) as amt1 from spark32_overwrite2 distribute by amt1)")
} finally {
  println("Location exists: " + fs.exists(new Path(loc)))
} {code}
Bug: Add an IDENTIFIER(stringLiteral) clause that maps a string to an identifier
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/23 22:36
Affects Version/s: 3.5.0

There is a requirement for SQL templates, where the table and or column names are provided through substitution. This can be done today using variable substitution:
SET hivevar:tabname = mytab;
SELECT * FROM ${ hivevar:tabname };

A straight variable substitution is dangerous since it does allow for SQL injection:
SET hivevar:tabname = mytab, someothertab;
SELECT * FROM ${ hivevar:tabname };

A way to get around this problem is to wrap the variable substitution with a clause that limits the scope t produce an identifier.
This approach is taken by Snowflake:
 [https://docs.snowflake.com/en/sql-reference/session-variables#using-variables-in-sql]

SET hivevar:tabname = 'tabname';
SELECT * FROM IDENTIFIER(${ hivevar:tabname })
Bug: Fix DatetimeIndex.microsecond to return 'int32' instead of 'int64' type of Index.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/23 03:59
Affects Version/s: 4.0.0

https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#index-can-now-hold-numpy-numeric-dtypes
Bug: Spark last window dont flush in append mode
Status: Open | Priority: Major | Resolution: 
Created: 01/Apr/23 07:52
Affects Version/s: 3.3.2

The problem is very simple, when you use *TUMBLING* *window* with {*}append mode{*}, then the window is closed +only when the next message arrives+ ({_}+watermark logic{_}). 

In the current implementation, if you *stop* *incoming* streaming data, the *last* window will *NEVER close* and we LOSE the last window data.

 

Business situation:

Worked correctly and new messages stop incoming and next message come in 5 hours  later and the client will get the message after 5 hours instead of the 10 seconds delay of window.

!https://user-images.githubusercontent.com/61819835/226478055-dc4a123c-4397-4eb0-b6ed-1e185b6fab76.png|width=707,height=294!

The current implementation needs to be improved. Include in spark internal mechanisms to close windows automatically.

 

*What we propose:*

Add third parameter 

{{{}DataFrame.{}}}{{{}withWatermark{}}}({_}eventTime{_}, {_}delayThreshold, *maxDelayClose*{_}). And then trigger will execute 
{code:java}
if(now - window.upper_bound > maxDelayClose){
     window.close().flush();
}
{code}
I assume it can be done in a day. It wasn't expected for us that our customers couldn't get the notifications. (the company is in the medical field).

 

simple code for problem:
{code:java}
kafka_stream_df = spark \
    .readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", KAFKA_BROKER) \
    .option("subscribe", KAFKA_TOPIC) \
    .option("includeHeaders", "true") \
    .load()

sel = (kafka_stream_df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
       .select(from_json(col("value").cast("string"), json_schema).alias("data"))
       .select("data.*")
       .withWatermark("dt", "1 seconds")
       .groupBy(window("dt", "10 seconds"))
       .agg(sum("price"))
      )
 
console = sel \
    .writeStream \
    .trigger(processingTime='10 seconds') \
    .format("console") \
    .outputMode("append")\
    .start()
{code}
Bug: RocksDB state store provider memory management enhancements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/23 18:24
Affects Version/s: 3.4.0

Today when RocksDB is used as a State Store provider, memory usage when writing using writeBatch is not capped. Also, a related issue is that the state store coordinator can create multiple RocksDB instances on a single node without enforcing a global limit on native memory usage. Due to these issues we could run into OOM issues and task failures. 
 
We are looking to improve this behavior by doing a series of improvements such as:
 * remove writeBatch and use native RocksDB operations
 * use writeBufferManager to manage global limit for all instances on a single node and accounting memtable + filter/index blocks usage as part of block cacheWith these changes we will be avoiding OOM issues around RocksDB native memory usage.
Bug: Inconsistency between from_avro and from_json function
Status: Open | Priority: Major | Resolution: 
Created: 19/Apr/23 18:47
Affects Version/s: 3.4.0

Spark from_avro function does not allow schema parameter to use dataframe column but takes only a String schema:
{code:java}
def from_avro(col: Column, jsonFormatSchema: String): Column {code}
This makes it impossible to deserialize rows of Avro records with different schema since only one schema string could be pass externally. 

 

Here is what I would expect like from_json function:
{code:java}
def from_avro(col: Column, jsonFormatSchema: Column): Column  {code}
code example:
{code:java}
import org.apache.spark.sql.functions.from_avro

val avroSchema1 = """{"type":"record","name":"myrecord","fields":[{"name":"str1","type":"string"},{"name":"str2","type":"string"}]}""" 

val avroSchema2 = """{"type":"record","name":"myrecord","fields":[{"name":"str1","type":"string"},{"name":"str2","type":"string"}]}"""


val df = Seq(
  (Array[Byte](10, 97, 112, 112, 108, 101, 49, 0), avroSchema1),
  (Array[Byte](10, 97, 112, 112, 108, 101, 50, 0), avroSchema2)
).toDF("binaryData", "schema")


val parsed = df.select(from_avro($"binaryData", $"schema").as("parsedData"))


parsed.show()


// Output:
// +------------+
// |  parsedData|
// +------------+
// |[apple1, 1.0]|
// |[apple2, 2.0]|
// +------------+
 {code}
Bug: Add option for applications to control saving of metadata in the External Shuffle Service LevelDB
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/23 05:22
Affects Version/s: 3.4.0

Currently, the External Shuffle Service stores application metadata in LevelDB. This is necessary to enable the shuffle server to resume serving shuffle data for an application whose executors registered before the NodeManager restarts. However, the metadata includes the application secret, which is stored in LevelDB without encryption. This is a potential security risk, particularly for applications with high security requirements. While filesystem access control lists (ACLs) can help protect keys and certificates, they may not be sufficient for some use cases. In response, we have decided not to store metadata for these high-security applications in LevelDB. As a result, these applications may experience more failures in the event of a node restart, but we believe this trade-off is acceptable given the increased security risk.
Bug: Support lz4raw compression codec for Parquet
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/23 06:59
Affects Version/s: 3.5.0

hadoop-parquet version should be updated to 1.3.0 (together with other parquet-mr libs)


{code:java}
java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 1 times, most recent failure: Lost task 2.0 in stage 1.0 (TID 3) (f2b63fdfa0a6 executor driver): java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZ4_RAW
    at java.base/java.lang.Enum.valueOf(Enum.java:273)
    at org.apache.parquet.hadoop.metadata.CompressionCodecName.valueOf(CompressionCodecName.java:26)
    at org.apache.parquet.format.converter.ParquetMetadataConverter.fromFormatCodec(ParquetMetadataConverter.java:636)
... {code}
Bug: Avoid allocation of unwritten ColumnVector in VectorizedReader
Status: Open | Priority: Major | Resolution: 
Created: 24/Apr/23 14:40
Affects Version/s: 3.4.1

Spark Vectorized Reader allocates the array for every fields for each value count even the array is ended up empty. This causes a high memory consumption when reading a table with large struct+array or many columns with sparse value. One way to fix this is by lazily allocating the column vector and only allocates the array only when it is needed (array is written).
Bug: Fix deserialisation issue when UDFs contain a lambda expression
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/23 05:08
Affects Version/s: 3.5.0

The following code:
{code:java}
class A(x: Int) { def get = x * 20 + 5 }
val dummyUdf = (x: Int) => new A(x).get
val myUdf = udf(dummyUdf)
spark.range(5).select(myUdf(col("id"))).as[Int].collect() {code}
hits the following error:
{noformat}
io.grpc.StatusRuntimeException: INTERNAL: cannot assign instance of java.lang.invoke.SerializedLambda to field ammonite.$sess.cmd26$Helper.dummyUdf of type scala.Function1 in instance of ammonite.$sess.cmd26$Helper
  io.grpc.Status.asRuntimeException(Status.java:535)
  io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
  org.apache.spark.sql.connect.client.SparkResult.org$apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:62)
  org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:114)
  org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:131)
  org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2687)
  org.apache.spark.sql.Dataset.withResult(Dataset.scala:3088)
  org.apache.spark.sql.Dataset.collect(Dataset.scala:2686)
  ammonite.$sess.cmd28$Helper.<init>(cmd28.sc:1)
  ammonite.$sess.cmd28$.<init>(cmd28.sc:7)
  ammonite.$sess.cmd28$.<clinit>(cmd28.sc){noformat}
Bug: Streaming progress struct (especially in Scala)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/23 17:45
Affects Version/s: 3.5.0

Streaming spark connect transfers streaming progress as full “json”.

This works ok for Python since it does not have any schema defined. 
But in Scala, it is a full fledged class. We need to decide if we want to match legacy Progress struct in spark-connect.
Bug: Cascade failure in Guava cache due to fate-sharing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/23 01:05
Affects Version/s: 3.4.0

Guava cache is widely used in spark, however, it suffers from fate-sharing behavior: If there are multiple requests trying to access the same key in the {{cache}} at the same time when the key is not in the cache, Guava cache will block all requests and create the object only once. If the creation fails, all requests will fail immediately without retry. So we might see task failure due to irrelevant failure in other queries due to fate sharing.

This fate sharing behavior might lead to unexpected results in some situation.

We can wrap around Guava cache with a KeyLock to synchronize all requests with the same key, so they will run individually and fail as if they come one at a time.
Bug: Update CBC mode in aes_encrypt()/aes_decrypt() to not use KDF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/23 18:55
Affects Version/s: 3.5.0

The current implementation of AES-CBC mode called via `{{{}aes_encrypt{}}}` and `{{{}aes_decrypt{}}}` uses a key derivation function (KDF) based on OpenSSL's [EVP_BytesToKey|https://www.openssl.org/docs/man3.0/man3/EVP_BytesToKey.html]. This is intended for generating keys based on passwords and OpenSSL's documents discourage its use: _"Newer applications should use a more modern algorithm"._

`{{{}aes_encrypt{}}}` and `{{{}aes_decrypt{}}}` should use the key directly in CBC mode, as it does for both GCM and ECB mode. The output should then be the initialization vector (IV) prepended to the ciphertext – as is done with GCM mode:
{{(16-byte randomly generated IV | AES-CBC encrypted ciphertext)}}
Bug: ArtifactManagerSuite can't run using maven
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/23 08:00
Affects Version/s: 3.5.0

run
{code:java}
build/mvn  clean install -DskipTests -Phive 
build/mvn test -pl connector/connect/server {code}
ArtifactManagerSuite failed due to 

 
{code:java}
23/04/26 16:00:07.666 ScalaTest-main-running-DiscoverySuite ERROR Executor: Could not find org.apache.spark.repl.ExecutorClassLoader on classpath! {code}
Bug: Dataset.observe is ignored when writing to Kafka with batch query
Status: Open | Priority: Major | Resolution: 
Created: 27/Apr/23 15:30
Affects Version/s: 3.3.2

When writing to Kafka with a batch query, metrics defined with {{Dataset.observe}} are not recorded. 

For example, 
{code:java}
import org.apache.spark.sql.execution.QueryExecution
import org.apache.spark.sql.util.QueryExecutionListener

spark.listenerManager.register(new QueryExecutionListener {
  override def onSuccess(funcName: String, qe: QueryExecution, durationNs: Long): Unit = {
    println(qe.observedMetrics)
  }

  override def onFailure(funcName: String, qe: QueryExecution, exception: Exception): Unit = {
    //pass
  }
})

val df = Seq(("k", "v")).toDF("key", "value")
val observed = df.observe("my_observation", lit("metric_value").as("some_metric"))
observed
  .write
  .format("kafka")
  .option("kafka.bootstrap.servers", "host1:port1")
  .option("topic", "topic1")
  .save()
{code}
prints {{{}Map(){}}}.
Bug: _metadata.file_path regression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/23 16:10
Affects Version/s: 3.4.0

As part of the [SparkPath refactor|https://issues.apache.org/jira/browse/SPARK-41970] the behavior of `_metadata.file_path` was inadvertently changed. In Spark 3.4+ it now returns a non-encoded path string, as opposed to a url-encoded path string.

This ticket is to fix that regression.
Bug: Improve TimestampNTZ support in JDBC data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/23 18:10
Affects Version/s: 3.3.3

[https://github.com/apache/spark/pull/36726] supports TimestampNTZ type in JDBC data source and [https://github.com/apache/spark/pull/37013] applies a fix to pass more test cases with H2.

The problem is that Java Timestamp is a poorly defined class and different JDBC drivers implement "getTimestamp" and "setTimestamp" with different expected behaviors in mind. The general conversion implementation would work with some JDBC dialects and their drivers but not others. This issue is discovered when testing with PostgreSQL database.

We will need to have dialect specific conversions between JDBC timestamps and TimestampNTZ.
Bug: Allow emitting zero values when deserializing protobuf messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/23 14:58
Affects Version/s: 3.4.0

Currently, when deserializing protobufs using from_protobuf, fields that are not explicitly present in the serialized message are deserialized as null in the resulting struct. However this includes singular proto3 scalars set explicitly to their default values, as they will [not appear in|https://protobuf.dev/programming-guides/field_presence/#presence-in-tag-value-stream-wire-format-serialization] the serialized protobuf.

For example, given a message format like

 
{code:java}
syntax = "proto3";
message Person {
string name = 1;
int64 age = 2;
optional string middle_name = 3;
optional int64 salary = 4;
}
{code}
and an example message like

 
{code:java}
SearchRequest(age = 0, middle_name = ""){code}
the result from calling from_protobuf on the serialized form of the above message would be

 
{code:java}
{"name": null, "age": null, "middle_name": "", "salary": null}{code}
 

It can be useful to deserialize these fields as their defaults, e.g.:

 
{code:java}
{"name": "", "age": 0, "middle_name": "", "salary": null}{code}
 

This behavior also exists in other major libraries, e.g.
 * java's jsonformat `includingDefaultValues` [https://protobuf.dev/reference/java/api-docs/com/google/protobuf/util/JsonFormat.Printer.html#includingDefaultValueFields--]
 * golang's jsonpb `emitDefaults` [https://pkg.go.dev/github.com/golang/protobuf/jsonpb#Marshaler]

I propose extending the spark-protobuf library to support this behavior.

 

PR: [https://github.com/apache/spark/pull/40686]
Bug: Correctness COUNT bug in correlated scalar subselect with `COUNT(*) is null`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/23 20:29
Affects Version/s: 3.4.0

Example query:
{code:java}
spark.sql("select *, (select (count(1)) is null from t1 where t0.a = t1.c) from t0").collect()
res6: Array[org.apache.spark.sql.Row] = Array([1,1.0,null], [2,2.0,false])  {code}
In this subquery, count(1) always evaluates to a non-null integer value, so count(1) is null is always false. The correct evaluation of the subquery is always false.

We incorrectly evaluate it to null for empty groups. The reason is that NullPropagation rewrites Aggregate [c] [isnull(count(1))] to Aggregate [c] [false] - this rewrite would be correct normally, but in the context of a scalar subquery it breaks our count bug handling in RewriteCorrelatedScalarSubquery.constructLeftJoins . By the time we get there, the query appears to not have the count bug - it looks the same as if the original query had a subquery with select any_value(false) from r..., and that case is _not_ subject to the count bug.

 

Postgres comparison show correct always-false result: [http://sqlfiddle.com/#!17/67822/5]

DDL for the example:
{code:java}
create or replace temp view t0 (a, b)
as values
    (1, 1.0),
    (2, 2.0);
create or replace temp view t1 (c, d)
as values
    (2, 3.0); {code}
Bug: DataFrame.toPandas with Arrow enabled should handle exceptions properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/23 01:32
Affects Version/s: 3.4.0

Currently {{DataFrame.toPandas}} doesn't capture exceptions happened in Spark properly.

{code:python}
>>> spark.conf.set("spark.sql.ansi.enabled", True)
>>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True)
>>> spark.sql("select 1/0").toPandas()
...
  An error occurred while calling o53.getResult.
: org.apache.spark.SparkException: Exception thrown in awaitResult:
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:322)
...
{code}
Bug: Add torch distributor data loader that loads data from spark partition data
Status: Resolved | Priority: Major | Resolution: Done
Created: 10/Apr/23 09:58
Affects Version/s: 3.5.0

Add torch distributor data loader that loads data from spark partition data.

 

We can add 2 APIs like:

Adds a `TorchDistributor` method API :
{code:java}
     def train_on_dataframe(self, train_function, spark_dataframe, *args, **kwargs):
        """
        Runs distributed training using provided spark DataFrame as input data.
        You should ensure the input spark DataFrame have evenly divided partitions,
        and this method starts a barrier spark job that each spark task in the job
        process one partition of the input spark DataFrame.
        Parameters
        ----------
        train_function :
            Either a PyTorch function, PyTorch Lightning function that launches distributed
            training. Note that inside the function, you can call
            `pyspark.ml.torch.distributor.get_spark_partition_data_loader` API to get a torch
            data loader, the data loader loads data from the corresponding partition of the
            input spark DataFrame.
        spark_dataframe :
            An input spark DataFrame that can be used in PyTorch `train_function` function.
            See `train_function` argument doc for details.
        args :
            `args` need to be the input parameters to `train_function` function. It would look like
            >>> model = distributor.run(train, 1e-3, 64)
            where train is a function and 1e-3 and 64 are regular numeric inputs to the function.
        kwargs :
            `kwargs` need to be the key-work input parameters to `train_function` function.
            It would look like
            >>> model = distributor.run(train, tol=1e-3, max_iter=64)
            where train is a function that has 2 arguments `tol` and `max_iter`.
        Returns
        -------
            Returns the output of `train_function` called with args inside spark rank 0 task.
        """{code}
 

Adds an loader API:

 
{code:java}
 def get_spark_partition_data_loader(num_samples, batch_size, prefetch=2):
    """
    This function must be called inside the `train_function` where `train_function`
    is the input argument of `TorchDistributor.train_on_dataframe`.
    The function returns a pytorch data loader that loads data from
    the corresponding spark partition data.
    Parameters
    ----------
    num_samples :
        Number of samples to generate per epoch. If `num_samples` is less than the number of
        rows in the spark partition, it generate the first `num_samples` rows of
        the spark partition, if `num_samples` is greater than the number of
        rows in the spark partition, then after the iterator loaded all rows from the partition,
        it wraps round back to the first row.
    batch_size:
        How many samples per batch to load.
    prefetch:
        Number of batches loaded in advance.
    """{code}