av9ash/GitBugs huggingface dataset:
Bug: OptimizeLocalShuffleReader causing data skew
Status: Open | Priority: Major | Resolution: 
Created: 31/Oct/22 07:49
Affects Version/s: 3.2.0

Because there are many empty files in the table, the partition num of OptimizeLocalShuffleReader to optimize partition num is 1

!image-2022-10-31-15-53-19-751.png!

!image-2022-10-31-15-57-41-599.png|width=1183,height=584!

!image-2022-10-31-15-50-36-435.png!

 

 

!image-2022-10-31-15-51-39-430.png!
Bug: Fix returning partial results in JSON data source and JSON functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/22 04:01
Affects Version/s: 3.4.0

I recently found an issue when parsing the following JSON file:
{code:java}
{"a": {"x": 1, "y": true}, "b": {"x": 1}}
{"a": {"x": 2}, "b": {"x": 2}}{code}
Trying to read such table with fixed schema where {{y}} is a struct column and not a boolean: 
{code:java}
val df = spark.read
  .schema("a struct<x: int, y: struct<x: int>>, b struct<x: int>")
  .json("path") {code}
results in the following answer:
{code:java}
a	                 b
null	                 null
{"x":2,"y":null}	{"x":2} {code}
Column {{b}} is valid and should be still parsed despite {{a}} having the wrong value.

 

This could be considered a follow-up to https://issues.apache.org/jira/browse/SPARK-33134.
Bug: Memory issue with Structured streaming
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/22 21:56
Affects Version/s: 3.2.2

In Pyspark Structured streaming with Kafka as source and sink, the driver as well as the executors seem to get OOM killed after a long period of time (few days). Not able to pinpoint to any specific thing.

But 8-12 hrs long runs also show the slow memory creep in Prometheus metrics values -
 # JVM Off-heap memory of both driver and executors keep on increasing over time (12-24hrs observation time) [I have NOT enabled off-heap usage]
 # JVM heap memory of executors also keeps on bumping up in slow steps.
 # JVM RSS of executors and driver keeps increasing but python RSS does not increase

-Basic operation of counting rows from within sdf.forEachBatch() is being done to debug ( -Original business logic has Some dropDuplicates, aggregations , windowing are being done within the forEachBatch.

-watermarking on a custom timestamp column is being done. 

 

Heap Dump analysis shows large no. of duplicate strings (which look like generated code). Further large no. of byte[], char[] and UTF8String objects.. Does this point to any potential memory leak in Tungsten optimizer related code?
Bug: The semantics of `F` in `DateTimeFormatter` have changed
Status: Open | Priority: Major | Resolution: 
Created: 13/Oct/22 12:04
Affects Version/s: 3.4.0

{code:java}
val createSql =
  """
    |create temporary view v as select col from values
    | (timestamp '1582-06-01 11:33:33.123UTC+080000'),
    | (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
    | (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
    | (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
    | (timestamp '2018-11-17 13:33:33.123Z'),
    | (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
    | (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') t(col)
    | """.stripMargin
sql(createSql)
withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> false.toString) {
  val rows = sql("select col, date_format(col, 'F') from v").collect()
  // scalastyle:off
  rows.foreach(println)
} {code}
 

Before Java 19, the result is 

 
{code:java}
[1582-05-31 19:40:35.123,3]
[1969-12-31 15:00:00.0,3]
[1970-12-31 04:59:59.999,3]
[1996-03-31 07:03:33.123,3]
[2018-11-17 05:33:33.123,3]
[2019-12-31 09:33:33.123,3]
[2100-01-01 01:33:33.123,1] {code}
Java 19

 
{code:java}
[1582-05-31 19:40:35.123,5]
[1969-12-31 15:00:00.0,5]
[1970-12-31 04:59:59.999,5]
[1996-03-31 07:03:33.123,5]
[2018-11-17 05:33:33.123,3]
[2019-12-31 09:33:33.123,5]
[2100-01-01 01:33:33.123,1] {code}
Bug: Allow Spark on K8s to integrate w/ Log Service
Status: In Progress | Priority: Major | Resolution: 
Created: 23/Oct/22 13:00
Affects Version/s: 3.4.0

https://docs.google.com/document/d/1MfB39LD4B4Rp7MDRxZbMKMbdNSe6V6mBmMQ-gkCnM-0/edit?usp=sharing
Bug: Use stable derived-column-alias algorithm, suitable for CREATE VIEW 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/22 18:08
Affects Version/s: 3.4.0

Spark has the ability derive column aliases for expressions if no alias was provided by the user.
E.g.
CREATE TABLE T(c1 INT, c2 INT);
SELECT c1, `(c1 + 1)`, c3 FROM (SELECT c1, c1 + 1, c1 * c2 AS c3 FROM T);

This is a valuable feature. However, the current implementation works by pretty printing the expression from the logical plan.  This has multiple downsides:
 * The derived names can be unintuitive. For example the brackets in `(c1 + 1)` or outright ugly, such as:
SELECT `substr(hello, 1, 2147483647)` FROM (SELECT substr('hello', 1)) AS T;
 * We cannot guarantee stability across versions since the logical lan of an expression may change.

The later is a major reason why we cannot allow CREATE VIEW without a column list except in "trivial" cases.

CREATE VIEW v AS SELECT c1, c1 + 1, c1 * c2 AS c3 FROM T;
Not allowed to create a permanent view `spark_catalog`.`default`.`v` without explicitly assigning an alias for expression (c1 + 1).

There are two way we can go about fixing this:
 # Stop deriving column aliases from the expression. Instead generate unique names such as `_col_1` based on their position in the select list. This is ugly and takes away the "nice" headers on result sets
 # Move the derivation of the name upstream. That is instead of pretty printing the logical plan we pretty print the lexer output, or a sanitized version of the expression as typed.
The statement as typed is stable by definition. The lexer is stable because i has no reason to change. And if it ever did we have a better chance to manage the change.

In this feature we propose the following semantic:
 # If the column alias can be trivially derived (some of these can stack), do so:
 ** a (qualified) column reference => the unqualified column identifier
cat.sch.tab.col => col
 ** A field reference => the fieldname
struct.field1.field2 => field2
 ** A cast(column AS type) => column
cast(col1 AS INT) => col1
 ** A map lookup with literal key => keyname
map.key => key
map['key'] => key
 ** A parameter less function => unqualified function name
current_schema() => current_schema
 # Take the lexer tokens of the expression, eliminate comments, and append them.
foo(tab1.c1 + /* this is a plus*/
1) => `foo(tab1.c1+1)`

 

Of course we wan this change under a config.
If the config is set we can allow CREATE VIEW to exploit this and use the derived expressions.

PS: The exact mechanics of formatting the name is very much debatable. E.g.spaces between token, squeezing out comments - upper casing - preserving quotes or double quotes...)
Bug: Add documentation (similar to Avro functions).
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/22 22:30
Affects Version/s: 3.3.0

Build documentation for protobufs similar to [https://spark.apache.org/docs/latest/sql-data-sources-avro.html]

This is a follow up from Protobuf PR https://github.com/apache/spark/pull/37972

cc: [~sanysandish@gmail.com] .
Bug: Spark will filter out data field sorting when dynamic partitions and data fields are sorted at the same time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/22 03:09
Affects Version/s: 3.1.2

When using dynamic partitions to write data and sort partitions and data fields, Spark will filter the sorting of data fields.

 

reproduce sql:
{code:java}
CREATE TABLE `sort_table`(
  `id` int,
  `name` string
  )
PARTITIONED BY (
  `dt` string)
stored as textfile
LOCATION 'sort_table';CREATE TABLE `test_table`(
  `id` int,
  `name` string)
PARTITIONED BY (
  `dt` string)
stored as textfile
LOCATION
  'test_table';//gen test data
insert into test_table partition(dt=20221011) select 10,"15" union all select 1,"10" union  all select 5,"50" union  all select 20,"2" union  all select 30,"14"  ;
set spark.hadoop.hive.exec.dynamici.partition=true;
set spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict;

// this sql sort with partition filed (`dt`) and data filed (`name`), but sort with `name` can not work
insert overwrite table sort_table partition(dt) select id,name,dt from test_table order by name,dt;
 {code}
 

The Sort operator of DAG has only one sort field, but there are actually two in SQL.(See the attached drawing)

 

It relate this issue : https://issues.apache.org/jira/browse/SPARK-40588
Bug: add named_struct to spark scala api for parity to spark sql
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/22 13:24
Affects Version/s: 3.3.0

To facilitate migration from cast(struct(.... to `named_struct` where comparison of structures is needed.

To maintain parity between APIs. Understand that using `expr` would work, but it wouldn't be typed.
Bug: Schema evolution for protobuf (and Avro too?)
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 04/Oct/22 21:02
Affects Version/s: 3.3.0

Protobuf & Avro should support schema evolution in streaming. We need to throw a specific error message when we detect newer version of the the schema in schema registry.

A couple of options for detecting version change at runtime:
 * How do we detect newer version from schema registry? It is contacted only during planning currently.

 * We could detect version id in coming messages.

 ** What if the id in the incoming message is newer than what our schema-registry reports after the restart?

 *** This indicates delayed syncs between customers schema-registry servers (should be rare). We can keep erroring out until it is fixed.

 *** Make sure we log the schema id used during planning.
Bug: Introduce window_time function to extract event time from the window column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/22 17:58
Affects Version/s: 3.4.0

Currently chaining of stateful operators is Spark Structured Streaming is not supported for various reasons and is blocked by the unsupported operations check (spark.sql.streaming.unsupportedOperationCheck flag). We propose to fix this as chaining of stateful operators is a common streaming scenario - e.g.

stream-stream join -> windowed aggregation

window aggregation -> window aggregation

etc

What is broken:
 # every stateful operator performs late record filtering against the global watermark. When chaining stateful operators (e.g. window aggregations) the output produced by the first stateful operator is effectively late against the watermark and thus filtered out by the next operator late record filtering (technically the next operator should not do late record filtering but it can be changed to assert for correctness detection, etc)
 # when chaining window aggregations, the first window aggregating operator produces records with schema \{ window: { start: Timestamp, end: Timestamp }, agg: Long } - there is not explicit event time in the schema to be used by the next stateful operator (the correct event time should be window.end - 1 )

 # stream-stream time-interval join can produce late records by semantics, e.g. if the join condition is:

left.eventTime BETWEEN right.eventTime + INTERVAL 1 HOUR right.eventTime - INTERVAL 1 HOUR

          the produced records can be delayed by 1 hr relative to the watermark.

Proposed fixes:

 1. 1 can be fixed by performing late record filtering against the previous microbatch watermark instead of the current microbatch watermark.

2. 2 can be fixed by allowing the window and session_window functions to work on the window column directly and compute the correct event time transparently to the user. Also, introduce window_time SQL function to compute correct event time from the window column.

3. 3 can be fixed by adding support for per-operator watermarks instead of a single global watermark. In the example of stream-stream time interval join followed by a stateful operator, the join operator will 'delay' the downstream operator watermarks by a correct value to handle the delayed records. Only stream-stream time-interval joins will be delaying the watermark, any other operators will not delay downstream watermarks.

 

*This ticket handles no. 2 of the proposal.* Others will be handled in separate ticket.
Bug: Make MSCK optional in MSCK REPAIR TABLE commands
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/22 20:41
Affects Version/s: 3.4.0

The current syntax for `MSCK REPAIR TABLE` is complex and difficult to understand. The proposal is to make the `MSCK` keyword optional so that `REPAIR TABLE` may be used in its stead.
Bug: JSON conversion of ArrayType is not properly supported in Spark 3.2/2.13
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/22 12:46
Affects Version/s: 3.2.0

In Spark 3.2 (Scala 2.13), values with {{ArrayType}} are no longer properly support with JSON; e.g.

{noformat}
import org.apache.spark.sql.SparkSession

case class KeyValue(key: String, value: Array[Byte])

val spark = SparkSession.builder().master("local[1]").appName("test").getOrCreate()

import spark.implicits._

val df = Seq(Array(KeyValue("foo", "bar".getBytes))).toDF()

df.foreach(r => println(r.json))
{noformat}

Expected:

{noformat}
[{foo, bar}]
{noformat}

Encountered:

{noformat}
java.lang.IllegalArgumentException: Failed to convert value ArraySeq([foo,[B@dcdb68f]) (class of class scala.collection.mutable.ArraySeq$ofRef}) with the type of ArrayType(Seq(StructField(key,StringType,false), StructField(value,BinaryType,false)),true) to JSON.
	at org.apache.spark.sql.Row.toJson$1(Row.scala:604)
	at org.apache.spark.sql.Row.jsonValue(Row.scala:613)
	at org.apache.spark.sql.Row.jsonValue$(Row.scala:552)
	at org.apache.spark.sql.catalyst.expressions.GenericRow.jsonValue(rows.scala:166)
	at org.apache.spark.sql.Row.json(Row.scala:535)
	at org.apache.spark.sql.Row.json$(Row.scala:535)
	at org.apache.spark.sql.catalyst.expressions.GenericRow.json(rows.scala:166)
{noformat}
Bug: Update embedded vis-timeline javascript resources
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/22 20:55
Affects Version/s: 3.3.0

Spark 3.3 currently ships with embedded vis-timeline javascript resources subject to CVE-2020-28487, detected as a minor problem by several static vulnerability assessment tools.

https://nvd.nist.gov/vuln/detail/CVE-2020-28487: 
bq. This affects the package vis-timeline before 7.4.4. An attacker with the ability to control the items of a Timeline element can inject additional script code into the generated application.

This issue is not meant to imply a security problem in Spark itself.
Bug: Keep removed executor info in decommission state
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/22 20:49
Affects Version/s: 3.4.0

Removed executor due to decommission should be kept in a separate set. To avoid OOM, set size will be limited to 1K or 10K.

FetchFailed caused by decom executor could be divided into 2 categories:
 # When FetchFailed reached DAGScheduler, the executor is still alive or is lost but the lost info hasn't reached TaskSchedulerImpl. This is already handled in SPARK-40979
 # FetchFailed is caused by decom executor loss, so the decom info is already removed in TaskSchedulerImpl. If we keep such info in a short period, it is good enough. Even we limit the size of removed executors to 10K, it could be only at most 10MB memory usage. In real case, it's rare to have cluster size of over 10K and the chance that all these executors decomed and lost at the same time would be small.
Bug: Both SparkSQL and DataFrame insert invalid DATE/TIMESTAMP as NULL
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 02/Oct/22 04:08
Affects Version/s: 3.2.1

h3. Describe the bug

When we construct a DataFrame with an invalid DATE/TIMESTAMP (e.g. {{{}1969-12-31 23:59:59 B{}}}) via {{{}spark-shell{}}}, or insert an invalid DATE/TIMESTAMP into a table via {{{}spark-sql{}}}, both interfaces unexpectedly evaluate the invalid value to {{{}NULL{}}}, instead of throwing an exception.
h3. To Reproduce

On Spark 3.2.1 (commit {{{}4f25b3f712{}}}), using {{{}spark-sql{}}}:
{code:java}
$SPARK_HOME/bin/spark-sql{code}
Execute the following:
{code:java}
spark-sql> create table timestamp_vals(c1 TIMESTAMP) stored as ORC;
spark-sql> insert into timestamp_vals select cast(" 1969-12-31 23:59:59 B "as timestamp);
spark-sql> select * from timestamp_vals;
NULL{code}
 
Using {{{}spark-shell{}}}:
{code:java}
$SPARK_HOME/bin/spark-shell{code}
 
Execute the following:
{code:java}
scala> import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.{Row, SparkSession}
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> val rdd = sc.parallelize(Seq(Row(Seq(" 1969-12-31 23:59:59 B ").toDF("time").select(to_timestamp(col("time")).as("to_timestamp")).first().getAs[java.sql.Timestamp](0))))
rdd: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = ParallelCollectionRDD[721] at parallelize at <console>:28
scala> val schema = new StructType().add(StructField("c1", TimestampType,  
true))
schema: org.apache.spark.sql.types.StructType = StructType(StructField(c1,TimestampType,true))
scala> val df = spark.createDataFrame(rdd, schema)
df194: org.apache.spark.sql.DataFrame = [c1: timestamp]
scala> df.show(false)
+----+
|c1  |
+----+
|null|
+----+
{code}
h3. Expected behavior

We expect both {{spark-sql}} & {{spark-shell}} interfaces to throw an exception for an invalid DATE/TIMESTAMP, like what they do for most of the other data types (e.g. invalid value {{"foo"}} for {{INT}} data type).
Bug: FLOAT/DOUBLE division by 0 gives Infinity/-Infinity/NaN in DataFrame but NULL in SparkSQL
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 02/Oct/22 00:15
Affects Version/s: 3.2.1

h3. Describe the bug

Storing a FLOAT/DOUBLE value with division by 0 (e.g. {{{}( 1.0/0 ).floatValue(){}}}) via {{spark-shell}} outputs {{{}Infinity{}}}. However, {{1.0/0}} ({{{}cast ( 1.0/0 as float){}}}) evaluated to {{NULL}} if the value is inserted into a FLOAT/DOUBLE column of a table via {{{}spark-sql{}}}.
h3. To Reproduce

On Spark 3.2.1 (commit {{{}4f25b3f712{}}}), using {{{}spark-sql{}}}:
{code:java}
$SPARK_HOME/bin/spark-sql{code}
Execute the following:
{code:java}
spark-sql> create table float_vals(c1 float) stored as ORC;
spark-sql> insert into float_vals select cast ( 1.0/0  as float);
spark-sql> select * from float_vals;
NULL{code}
 

Using {{{}spark-shell{}}}:
{code:java}
$SPARK_HOME/bin/spark-shell{code}
Execute the following:
{code:java}
scala> import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.{Row, SparkSession}
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> val rdd = sc.parallelize(Seq(Row(( 1.0/0 ).floatValue())))
rdd: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = ParallelCollectionRDD[180] at parallelize at <console>:28
scala> val schema = new StructType().add(StructField("c1", FloatType, true) 
)
schema: org.apache.spark.sql.types.StructType = StructType( StructField(c1,FloatType,true))
scala> val df = spark.createDataFrame(rdd, schema)
df: org.apache.spark.sql.DataFrame = [c1: float]
scala> df.show(false)
+---------+
|c1       |
+---------+
|Infinity |
+---------+
{code}
h3. Expected behavior

We expect the two Spark interfaces ({{{}spark-sql{}}} & {{{}spark-shell{}}}) to behave consistently for the same data type & input combination & configuration ({{{}FLOAT/DOUBLE{}}} and {{{}1.0/0{}}}).
Bug: ExtractGenerator sets incorrect nullability in new Project
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/22 23:13
Affects Version/s: 3.1.3

Example:
{noformat}
select c1, explode(c4) as c5 from (
  select c1, array(c3) as c4 from (
    select c1, explode_outer(c2) as c3
    from values
    (1, array(1, 2)),
    (2, array(2, 3)),
    (3, null)
    as data(c1, c2)
  )
);

+---+---+
|c1 |c5 |
+---+---+
|1  |1  |
|1  |2  |
|2  |2  |
|2  |3  |
|3  |0  |
+---+---+
{noformat}
In the last row, {{c5}} is 0, but should be {{NULL}}.

Another example:
{noformat}
select c1, exists(c4, x -> x is null) as c5 from (
  select c1, array(c3) as c4 from (
    select c1, explode_outer(c2) as c3
    from values
    (1, array(1, 2)),
    (2, array(2, 3)),
    (3, null)
    as data(c1, c2)
  )
);

+---+-----+
|c1 |c5   |
+---+-----+
|1  |false|
|1  |false|
|2  |false|
|2  |false|
|3  |false|
+---+-----+
{noformat}
In the last row, {{false}} should be {{true}}.

In both cases, at the time {{CreateArray(c3)}} is instantiated, {{c3}}'s nullability is incorrect because the new projection created by {{ExtractGenerator}} uses {{generatorOutput}} from {{explode_outer(c2)}} as a projection list. {{generatorOutput}} doesn't take into account that {{explode_outer(c2)}} is an _outer_ explode, so the nullability setting is lost.

{{UpdateAttributeNullability}} will eventually fix the nullable setting for attributes referring to {{c3}}, but it doesn't fix the {{containsNull}} setting for {{c4}} in {{explode(c4)}} (from the first example) or {{exists(c4, x -> x is null)}} (from the second example).

This example fails with a {{NullPointerException}}:
{noformat}
select c1, inline_outer(c4) from (
  select c1, array(c3) as c4 from (
    select c1, explode_outer(c2) as c3
    from values
    (1, array(named_struct('a', 1, 'b', 2))),
    (2, array(named_struct('a', 3, 'b', 4), named_struct('a', 5, 'b', 6))),
    (3, null)
    as data(c1, c2)
  )
);

22/10/27 11:53:20 ERROR Executor: Exception in task 1.0 in stage 1.0 (TID 2)
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.generate_doConsume_1$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.generate_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:364)
{noformat}