av9ash/GitBugs huggingface dataset:
Bug: Hitting Interrupt when no cell is running kills pyspark jupyter kernel
Status: Open | Priority: Major | Resolution: 
Created: 29/Jul/22 05:19
Affects Version/s: 3.2.2

In Pyspark 3.2 kernel with Jupyter notebook, When no cell is running and I hit interrupt kernel, it actually kills pyspark kernel, where it's supposed to do NOTHING.|

I defined one startup script and in that startup script, If I override default interrupt signal handler to SIGIGN, it works.
{quote} signal(SIGINT, SIG_IGN)
{quote}
Are users of pyspark need to do this step explicity? I wonder why it doesn't work by default
Bug: Organize the check of push down information for JDBCV2Suite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/22 09:38
Affects Version/s: 3.4.0

Currently, JDBCV2Suite have many test cases check the push-down information looks not clean.
For example,

{code:java}
checkPushedInfo(df,
  "PushedFilters: [DEPT IS NOT NULL, DEPT > 1], PushedLimit: LIMIT 1, ")
{code}

If we change it to below looks better.

{code:java}
checkPushedInfo(df,
  "PushedFilters: [DEPT IS NOT NULL, DEPT > 1]",
  "PushedLimit: LIMIT 1")
{code}
Bug: Support recursive references in Avro schemas in Spark
Status: Open | Priority: Major | Resolution: 
Created: 18/Jul/22 00:50
Affects Version/s: 3.4.0

This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718. 

It would be great if Spark could support recursive references in Avro schemas as currently this is disabled and the following exception is thrown: 
{code:java}
org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive reference in Avro schema, which can not be processed by Spark: {
  "type": "record",
  "name": "Struct",
  "fields": [
    {
      "name": "fields",
      "type": [
        "null",
        {
          "type": "array",
          "items": {
            "type": "record",
            "name": "fields",
            "fields": [
              {
                "name": "value",
                "type": [
                  "null",
                  "string"
                ],
                "default": null
              }
            ]
          }
        }
      ],
      "default": null
    }
  ]
}{code}
I would appreciate it if someone could comment on the overall effort to make recursive Avro schemas work in Spark and whether or not there are any API restrictions that could potentially hinder the implementation, thanks.
Bug: Push limit 1 to the aggregate's child plan if grouping expressions and aggregate expressions are foldable
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Jul/22 08:28
Affects Version/s: 3.4.0

If all group expressions are foldable, the result of this aggregate will always be OneRowRelation.
And if all aggregate expressions are foldable, we can add limit 1 to the child plan.

For example:

Table *tab* (it can have data or empty): 
{code:sql}
create table tab(key int, value string) using parquet
{code}

Query : 
{code:sql}
SELECT distinct 1001 as id , cast('2022-06-03' as date) AS DT 
FROM tab
{code}
is equal to 
{code:sql}
SELECT distinct 1001 as id , cast('2022-06-03' as date) AS DT 
FROM (
    SELECT *
    FROM tab
    LIMIT 1
)
{code}

Query:
{code:sql}
SELECT 1001 AS id, cast('2022-06-03' as date) AS DT 
FROM tab
group by 'x'
{code}
is equal to 
{code:sql}
SELECT 1001 AS id, cast('2022-06-03' as date) AS DT 
FROM (
    SELECT *
    FROM tab
    LIMIT 1
)
group by 'x'
{code}
Bug: Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/22 07:19
Affects Version/s: 3.1.0

The {{UnsafeRow}} structural integrity check in {{UnsafeRowUtils.validateStructuralIntegrity}} is added in Spark 3.1.0. It’s supposed to validate that a given {{UnsafeRow}} conforms to the format that the {{UnsafeRowWriter}} would have produced.

Currently the check expects all fields that are marked as null should also have its field (i.e. the fixed-length part) set to all zeros. It needs to be updated to handle a special case for variable-length {{{}Decimal{}}}s, where the {{UnsafeRowWriter}} may mark a field as null but also leave the fixed-length part of the field as {{OffsetAndSize(offset=current_offset, size=0)}}. This may happen when the {{Decimal}} being written is either a real {{null}} or has overflowed the specified precision.

Logic in {{UnsafeRowWriter}}:

in general:
{code:scala}
  public void setNullAt(int ordinal) {
    BitSetMethods.set(getBuffer(), startingOffset, ordinal); // set null bit
    write(ordinal, 0L);                                      // also zero out the fixed-length field
  } {code}
special case for {{DecimalType}}:
{code:scala}
      // Make sure Decimal object has the same scale as DecimalType.
      // Note that we may pass in null Decimal object to set null for it.
      if (input == null || !input.changePrecision(precision, scale)) {
        BitSetMethods.set(getBuffer(), startingOffset, ordinal); // set null bit
        // keep the offset for future update
        setOffsetAndSize(ordinal, 0);                            // doesn't zero out the fixed-length field
      } {code}
The special case is introduced to allow all {{DecimalType}}s (including both fixed-length and variable-length ones) to be mutable – thus need to leave space for the variable-length field even if it’s currently null.

Note that this special case in {{UnsafeRowWriter}} has been there since Spark 1.6.0, where as the integrity check was added in Spark 3.1.0. The check was originally added for Structured Streaming’s checkpoint evolution validation, so that a newer version of Spark can check whether or not an older checkpoint file for Structured Streaming queries can be supported, and/or if the contents of the checkpoint file is corrupted.
Bug: 0.0 and -0.0 are not consistent in set operations 
Status: Open | Priority: Major | Resolution: 
Created: 23/Jul/22 00:10
Affects Version/s: 3.2.1

This is a continuation of the issue described in SPARK-32110.

When using Array set-based functions {{array_union}}, {{array_intersect}}, {{array_except}} and {{arrays_overlap}}, {{0.0}} and {{-0.0}} have inconsistent behavior.

When parsed, {{-0.0}} is normalized to {{0.0}}. Therefore if I use {{array_union}} for example with these values directly, {{array(-0.0)}} becomes {{array(0.0)}}. See the example below using {{array_union}}:

{code:java}
scala> val df = spark.sql("SELECT array_union(array(0.0), array(-0.0))")
df: org.apache.spark.sql.DataFrame = [array_union(array(0.0), array(0.0)): array<decimal(1,1)>]
scala> df.collect()
res2: Array[org.apache.spark.sql.Row] = Array([WrappedArray(0.0)])
{code}

In this case, {{0.0}} and {{-0.0}} are considered equal and the union of the arrays produces a single value: {{0.0}}.

However, if I try this operation using a constructed dataframe, these values are not equal, and the result is an array with both {{0.0}} and {{-0.0}}.

{code:java}
scala> val df = List((Array(0.0), Array(-0.0))).toDF("a", "b")
df: org.apache.spark.sql.DataFrame = [a: array<double>, b: array<double>]

scala> df.selectExpr("array_union(a, b)").collect()
res3: Array[org.apache.spark.sql.Row] = Array([WrappedArray(0.0, -0.0)])
{code}

For {{arrays_overlap}}, here is a similar version of that inconsistency:

{code:java}
scala> val df = spark.sql("SELECT arrays_overlap(array(0.0), array(-0.0))")
df: org.apache.spark.sql.DataFrame = [arrays_overlap(array(0.0), array(0.0)): boolean]

scala> df.collect
res4: Array[org.apache.spark.sql.Row] = Array([true])
{code}

{code:java}
scala> val df = List((Array(0.0), Array(-0.0))).toDF("a", "b")
df: org.apache.spark.sql.DataFrame = [a: array<double>, b: array<double>]

scala> df.selectExpr("arrays_overlap(a, b)")
res5: org.apache.spark.sql.DataFrame = [arrays_overlap(a, b): boolean]

scala> df.selectExpr("arrays_overlap(a, b)").collect
res6: Array[org.apache.spark.sql.Row] = Array([false])
{code}

It looks like this is due to the fact that in the constructed dataframe case, the Double value is hashed by using {{java.lang.Double.doubleToLongBits}}, which will treat {{0.0}} and {{-0.0}} as distinct because of the sign bit.

See here for more information: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala#L312-L321

I can also confirm that the same behavior occurs with FloatType and the use of {{java.lang.Float.floatToIntBits}}
Bug: Behavior differs between arrays_overlap and array_contains for negative 0.0
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/22 22:37
Affects Version/s: 3.2.2

{{array_contains([0.0], -0.0)}} will return true. {{array_overlaps([0.0], [-0.0])}} will return false. I think we generally want to treat -0.0 and 0.0 as the same (see https://github.com/apache/spark/blob/e9eb28e27d10497c8b36774609823f4bbd2c8500/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/SQLOrderingUtil.scala#L28)
However, the {{Double::equals}} method doesn't. Therefore, we should either mark double as false in [TypeUtils#typeWithProperEquals|https://github.com/apache/spark/blob/e9eb28e27d10497c8b36774609823f4bbd2c8500/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TypeUtils.scala#L96], or we should wrap it with our own equals method that handles this case.

Java code snippets showing the issue:

{code:java}
dataset = sparkSession.createDataFrame(
            List.of(RowFactory.create(List.of(-0.0))),
            DataTypes.createStructType(ImmutableList.of(DataTypes.createStructField(
                    "doubleCol", DataTypes.createArrayType(DataTypes.DoubleType), false))));
        Dataset<Row> df = dataset.withColumn(
            "overlaps", functions.arrays_overlap(functions.array(functions.lit(+0.0)), dataset.col("doubleCol")));
        List<Row> result = df.collectAsList(); // [[WrappedArray(-0.0),false]]
{code}

{code:java}
dataset = sparkSession.createDataFrame(
                List.of(RowFactory.create(-0.0)),
                DataTypes.createStructType(
                        ImmutableList.of(DataTypes.createStructField("doubleCol", DataTypes.DoubleType, false))));
        Dataset<Row> df = dataset.withColumn(
                "contains", functions.array_contains(functions.array(functions.lit(+0.0)), dataset.col("doubleCol")));
        List<Row> result = df.collectAsList(); // [[-0.0,true]]
{code}
Bug: Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/22 03:02
Affects Version/s: 3.4.0

With SPARK-39748, Spark includes the origin logical plan for LogicalRDD if it comes from DataFrame, to achieve carrying-over stats as well as providing information to possibly connect two disconnected logical plans into one.

After we introduced the change, we figured out several issues:

1. One of major use case for DataFrame.checkpoint is ML, especially "iterative algorithm", which purpose is to "prune" the logical plan. That is against the purpose of including origin logical plan and we have a risk to have nested LogicalRDDs which grows the size of logical plan infinitely.

2. We leverage logical plan to carry over stats, but the correct stats information is in optimized plan.

3. (Not an issue but missing spot) constraints is also something we can carry over.

To address above issues, it would be better if we include stats and constraints in LogicalRDD rather than logical plan.
Bug: [SparkHistoryServer] Rewrite JMXJsonServlet to work with the current version of Spark
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/22 11:59
Affects Version/s: 2.3.4

When i want to collect the details of SparkHistoryServer, such as GC, memory. But i cant find a way. I need to expose an interface for the other exporter to collect them.

I find hadoop just do it, but it conflict with spark. So i rewrite it in scala.
Bug: Migrate melt function in Pandas API to PySpark / Scala unpivot
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/22 13:15
Affects Version/s: 3.4.0

With SPARK-38864 the Scala Dataset API now has an unpivot / melt function. The existing melt function in Pandas API should use that implementation internally, or call into the PySpark unpivot (SPARK-39877).
Bug: Race condition related to interruption of task threads while they are in RocksDBLoader.loadLibrary()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/22 01:42
Affects Version/s: 3.2.0

One of our workloads experienced a rare failure in `RocksDBLoader`
{code:java}
Caused by: java.lang.IllegalThreadStateException
	at java.lang.Thread.start(Thread.java:708)
	at org.apache.spark.sql.execution.streaming.state.RocksDBLoader$.loadLibrary(RocksDBLoader.scala:51) {code}
After investigation, we determined that was due to task cancellation: if the task which starts the RocksDB library loading is interrupted, another thread may begin a load and crash with the thread state exception.

Skimming through the code in [RocksDBLoader|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBLoader.scala], I spot a potential race condition:
 * Although the native JNI call is uninterruptible, the thread which calls loadLibrary is still interruptible. Let’s call that thread the “task thread”.

 * Say we have two tasks, A and B, which both want to load the JNI library.

 * Say that Task A wins the race to perform the load and enters the synchronized block in loadLibrary(), spawns a child thread to perform the actual loading, then blocks in the loadLibraryThread.join() call.

 * If Task A is interrupted, an InterruptedException will be thrown and it will exit the loadLibrary synchronized block.

 * At this point, Task B enters the synchronized block and sees that exception == null because the loading thread is still running, so it calls loadLibraryThread.start() and hits the thread state error.

One way to fix this is to add
{code:java}
 if (loadLibraryThread.getState == Thread.State.NEW) {
        loadLibraryThread.start()
      }{code}
to ensure that only one thread starts the loadLibraryThread. If the original starter thread is interrupted then a new thread will encounter this block, skip the start(), proceed to the join() and block on the original load thread.

I will submit a PR with this fix.
Bug: Inconsistent parsing of unusually encoded CSV
Status: Open | Priority: Major | Resolution: 
Created: 22/Jul/22 11:56
Affects Version/s: 3.2.2

MCVE with PySpark code (but inherently in Spark itself, not PySpark):
{code:java}
spark = SparkSession.builder.appName("...").getOrCreate()

min_schema = StructType(
    [
        StructField("dummy_col", StringType(), True),
        StructField("record_id", IntegerType(), nullable=False),
        StructField("dummy_after", StringType(), nullable=False),
    ]
)


df = (
    spark.read.option("mode", "FAILFAST")
    .option("quote", '"')
    .option("escape", '"')
    .option("inferSchema", "false")
    .option("multiline", "true")
    .option("ignoreLeadingWhiteSpace", "true")
    .option("ignoreTrailingWhiteSpace", "true")
    .schema(min_schema)
    .csv(f'min_repro.csv', header=True)
) {code}
min_repro.csv:
{code:java}
dummy_col,record_id,dummy_after
"",1,", Unusual value with comma included"
B,2,"Unusual value with escaped quote and comma ""like, this" {code}
Spark behaves inconsistently

 

1. Collect works fine:
{code:java}
df.collect()

[Row(dummy_col=None, record_id=1, dummy_after=', Unusual value with comma included'),
Row(dummy_col='B', record_id=2, dummy_after='Unusual value with escaped quote and comma "like, this')] {code}
2. Trivial query on same DF fails
{code:java}
if df.count() != df.select('record_id').distinct().count():
    pass {code}
Error:
{code:java}
Py4JJavaError: An error occurred while calling o357.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 1 times, most recent failure: Lost task 0.0 in stage 17.0 (TID 13, localhost, executor driver): org.apache.spark.SparkException: Malformed records are detected in record parsing. Parse Mode: FAILFAST.
...
Caused by: java.lang.NumberFormatException: For input string: "Unusual value with comma included""
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) {code}
 

I understand that input file encoding is unusual, but Spark still shouldn't parse it differently between one or other dataframe method.
Bug: Passing an empty Metadata object to Column.as() should clear the metadata
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Jul/22 05:54
Affects Version/s: 3.3.0

h2. Description

The Spark DataFrame API allows developers to attach arbiotrary metadata to individual columns as key/value pairs. The attachment is performed via the method "Column.as(name, metadata)". This works as expected, as long as the metadata object is not empty. But when passing an empty metadata object, the final column in the resulting DataFrame will still hold the metadata of the original incoming column, i.e. you cannot use this method to essentially reset the metadata of a column.

This is not the expected behaviour and has changed in Spark 3.3.0. In Spark 3.2.1 and earlier, passing an empty metadata object to the method "Column.as(name, metadata)" resets the columns metadata as expected.

h2. Steps to Reproduce

The following code snippet will show the issue in Spark shell:
{code:scala}
import org.apache.spark.sql.types.MetadataBuilder

// Create a DataFrame with one column with Metadata attached
val df1 = spark.range(1,10)
    .withColumn("col_with_metadata", col("id").as("col_with_metadata", new MetadataBuilder().putString("metadata", "value").build()))

// Create a derived DataFrame which should reset the metadata of the column
val df2 = df1.select(col("col_with_metadata").as("col_without_metadata", new MetadataBuilder().build()))

// Display metadata of both DataFrames columns
println(s"df1 metadata: ${df1.schema("col_with_metadata").metadata}")
println(s"df2 metadata: ${df2.schema("col_without_metadata").metadata}")
{code} 

This code results in the following lines printed onto the console
{code}
df1 metadata: {"metadata":"value"}
df2 metadata: {"metadata":"value"}
{code}

This result does not meet my expectations. I expect that df1 has non-empty metadata, but df2 should have empty metadata. But this is not the case, df2 still holds the same metadata as df1.

h2. Analysis

I think the problem stems from the changes in the method "trimNonTopLevelAliases" in the class AliasHelper:
{code:scala}
  protected def trimNonTopLevelAliases[T <: Expression](e: T): T = {
    val res = e match {
      case a: Alias =>
        val metadata = if (a.metadata == Metadata.empty) {
          None
        } else {
          Some(a.metadata)
        }
        a.copy(child = trimAliases(a.child))(
          exprId = a.exprId,
          qualifier = a.qualifier,
          explicitMetadata = metadata,
          nonInheritableMetadataKeys = a.nonInheritableMetadataKeys)
      case a: MultiAlias =>
        a.copy(child = trimAliases(a.child))
      case other => trimAliases(other)
    }

    res.asInstanceOf[T]
  }
{code}

The method will remove any empty metadata object from an Alias, which in turn means that Alias will inherit its childs metadata.
Bug: add_months() returns a java error on overflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/22 12:41
Affects Version/s: 3.4.0

The code below throws an Java exception, see
{code:java}
spark.sql("SET spark.sql.ansi.enabled=true").show()spark.sql("SELECT add_months('5500000-12-31', 10000000)").show()java.lang.ArithmeticException: integer overflow  at java.base/java.lang.Math.toIntExact(Math.java:1074)  at org.apache.spark.sql.catalyst.util.DateTimeUtils$.localDateToDays(DateTimeUtils.scala:550)  at org.apache.spark.sql.catalyst.util.DateTimeUtils$.dateAddMonths(DateTimeUtils.scala:736) {code}
but it should throw Spark's exception w/ an error class.
Bug: Deprecate Trigger.Once and Promote Trigger.AvailableNow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/22 06:44
Affects Version/s: 3.4.0

Quoting the discussion in spark dev@: [link|https://lists.apache.org/thread/2xnxlxhw245cmspd8nd17cq5doj2c7hc]

Rationalization:

The expected behavior of Trigger.Once is like reading all available data after the last trigger and processing them. This holds true when the last run was gracefully terminated, but there are cases streaming queries to not be terminated gracefully. There is a possibility the last run may write the offset for the new batch before termination, then a new run of Trigger.Once only processes the data which was built in the latest unfinished batch and doesn't process new data.

The behavior is not deterministic from the users' point of view, as end users wouldn't know whether the last run wrote the offset or not, unless they look into the query's checkpoint by themselves.

While Trigger.AvailableNow came to solve the scalability issue on Trigger.Once, it also ensures that it tries to process all available data at the point of time it is triggered, which consistently works as expected behavior of Trigger.Once.

Another issue on Trigger.Once is that it does not trigger a no-data batch immediately. When the watermark is calculated in batch N, it takes effect in batch N + 1. If the query is scheduled to be run per day, you can see the output from the new watermark in the query run the next day. Thanks to the behavior of Trigger.AvailableNow, it handles no-data batch as well before termination of the query.

There was no strong feedback in the discussion thread, but accounting the fact we have very small number of contributors (including committers/PMC members) being active in SS area, we have to just go with lazy consensus.
Bug: Fix PushDownLeftSemiAntiJoin push through project
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Jul/22 09:42
Affects Version/s: 3.4.0

Push LeftSemi/LeftAnti through project is not always effective

1. project is used to do column pruning, so it conflicts with ColumnPruning
2. if the project contains complex expression and join condition reference it, it can be regression since the complex expression will be evaluated more than once
Bug: Add DecimalDivideWithOverflowCheck for decimal average
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/22 08:03
Affects Version/s: 3.4.0

If the result data type is decimal, the Average will first calculate the result using the default precison and scale of divide, then cast to the result data type. We should do calculate and return the result data type directly so that we can avoid the precision loss. It can also save one unnecessary cast.
Bug: Queries accessing METADATA struct crash on partitioned tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/22 07:58
Affects Version/s: 3.3.0

There is a problem with a projection we use in `FileScanRDD` to join the metadata row to the row produced by the reader.

https://github.com/apache/spark/blob/e4ca8424474e571d8e137388fe5d54732b68c2f3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala#L128-L133

The issue is that the projection omits partition columns. As a result, the expressions down the line return a malformed row. The errors crash the query, but the exact message can vary (for example: failed assertion on number of fields in the row, accessing field of incorrect type).

This defect affects only readers producing rows (as opposed to batches), and only data sets using dynamic partitioning.