av9ash/GitBugs huggingface dataset:
Bug: Custom metrics should be updated after commit too
Status: Open | Priority: Minor | Resolution: 
Created: 01/Nov/23 12:57
Affects Version/s: 3.4.1

We have a DataWriter component, which processes records in configurable batches, which are accumulated in {{write(T record)}} implementation and sent to the persistent store when the configured batch size is reached. Within this approach, last batch is handled during {{commit()}} call, as there is no other mechanism of knowing if there are more records or not.

We are now adding support for custom metrics, by implementing the {{supportedCustomMetrics()}} and {{currentMetricsValues()}} in the {{Write}} and {{DataWriter}} implementations. The problem we see is, since {{CustomMetrics.updateMetrics}} is only called [during|https://github.com/apache/spark/blob/af8907a0873f5ca192b150f28a0c112107594722/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala#L443-L443] and [just after|https://github.com/apache/spark/blob/af8907a0873f5ca192b150f28a0c112107594722/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala#L451-L451] record processing, we do not observe the complete metrics since the last batch that is handled during {{commit()}} call is not collected/updated.

We propose to also to add {{CustomMetrics.updateMetrics}} call after {{commit()}} is processed successfully, ideally just before {{run}} function exits (maybe just above [https://github.com/apache/spark/blob/af8907a0873f5ca192b150f28a0c112107594722/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala#L473-L473]).
Bug: Deprecate DStream API
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 15/Jan/23 23:38
Affects Version/s: 3.4.0

We've got consensus to deprecate DStream API in 3.4.0.

[https://lists.apache.org/thread/342qnjxmnoydzxr0k8yq4roxdmqjhw9x]

This Jira ticket is to track the effort of action items:
 * Add "deprecation" annotation to the user-facing public API in streaming directory (DStream)
 * Write a release note to explicitly mention the deprecation. (Maybe promote again that they are encouraged to move to SS.)
Bug: Document work (Release note & Guide doc) for SPARK-40925
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 18/Jan/23 07:48
Affects Version/s: 3.4.0

SPARK-40925 fixed the bug which introduced the major limitation we described in the guide doc.

[https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#limitation-of-global-watermark]

Many limitations described in the guide doc are resolved by SPARK-40925. We even unblocked the functionality by SPARK-40940, so the doc is out of sync with the codebase.

We probably should update the guide doc to describe a new limitation. We also have to write down release note for SPARK-40925 as well.
Bug: Tasks are over-scheduled with TaskResourceProfile
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Jan/23 02:44
Affects Version/s: 3.4.0

{code:java}
test("SPARK-XXX") {
  val conf = new SparkConf().setAppName("test").setMaster("local-cluster[1,4,1024]")
  sc = new SparkContext(conf)
  val req = new TaskResourceRequests().cpus(3)
  val rp = new ResourceProfileBuilder().require(req).build()

  val res = sc.parallelize(Seq(0, 1), 2).withResources(rp).map { x =>
    Thread.sleep(5000)
    x * 2
  }.collect()
  assert(res === Array(0, 2))
} {code}
In this test, tasks are supposed to be scheduled in order since each task requires 3 cores but the executor only has 4 cores. However, we noticed 2 tasks are launched concurrently from the logs.

It turns out that we used the TaskResourceProfile (taskCpus=3) of the taskset for task scheduling:
{code:java}
val rpId = taskSet.taskSet.resourceProfileId
val taskSetProf = sc.resourceProfileManager.resourceProfileFromId(rpId)
val taskCpus = ResourceProfile.getTaskCpusOrDefaultForProfile(taskSetProf, conf) {code}
but the ResourceProfile (taskCpus=1) of the executor for updating the free cores in ExecutorData:
{code:java}
val rpId = executorData.resourceProfileId
val prof = scheduler.sc.resourceProfileManager.resourceProfileFromId(rpId)
val taskCpus = ResourceProfile.getTaskCpusOrDefaultForProfile(prof, conf)
executorData.freeCores -= taskCpus {code}
which results in the inconsistency of the available cores.
Bug: Fix a correctness bug in existence DEFAULT value lookups for the Orc data source
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Jan/23 18:42
Affects Version/s: 3.4.0

This issue aims to fix a correctness bug in existence DEFAULT value lookups during data source scans

Initially, SPARK-39862 tried to fix this, but was reverted via SPARK-41858 due to the perf issue.
Bug: Upgrade Parquet to fix off-heap memory leaks in Zstd codec
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 09/Jan/23 23:48
Affects Version/s: 3.1.3

Recently, native memory leak have been discovered in Parquet in conjunction of it using Zstd decompressor from luben/zstd-jni library (PARQUET-2160).

This is very problematic to a point where we can't use Parquet w/ Zstd due to pervasive OOMs taking down our executors and disrupting our jobs.

Luckily fix addressing this had already landed in Parquet:
[https://github.com/apache/parquet-mr/pull/982]

 

Now, we just need to
 # Updated version of Parquet is released in a timely manner
 # Spark is upgraded onto this new version in the upcoming release
Bug: When the inserted partition type is of string type, similar `dt=01` will be converted to `dt=1`
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 11/Jan/23 09:12
Affects Version/s: 3.4.0

At present, during the process of upgrading Spark2.4 to Spark3.2, we carefully read the migration documentwe and found a kind of situation not involved:
{code:java}
create table if not exists test_90(a string, b string) partitioned by (dt string);
desc formatted test_90;
// case1
insert into table test_90 partition (dt=05) values("1","2");
// case2
insert into table test_90 partition (dt='05') values("1","2");
drop table test_90;{code}
in spark2.4.3, it will generate such a path:
{code:java}
// the path
hdfs://test5/user/hive/db1/test_90/dt=05 

//result
spark-sql> select * from test_90;
1       2       05
1       2       05
Time taken: 1.316 seconds, Fetched 2 row(s)

spark-sql> show partitions test_90; 
dt=05 
Time taken: 0.201 seconds, Fetched 1 row(s)

spark-sql> select * from test_90 where dt='05';
1       2       05
1       2       05
Time taken: 0.212 seconds, Fetched 2 row(s)

spark-sql> explain insert into table test_90 partition (dt=05) values("1","2");
== Physical Plan ==
Execute InsertIntoHiveTable InsertIntoHiveTable `db1`.`test_90`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, Map(dt -> Some(05)), false, false, [a, b]
+- LocalTableScan [a#116, b#117]
Time taken: 1.145 seconds, Fetched 1 row(s){code}
in spark3.2.0, it will generate two path:
{code:java}
// the path
hdfs://test5/user/hive/db1/test_90/dt=05 
hdfs://test5/user/hive/db1/test_90/dt=5 

// result
spark-sql> select * from test_90;
1       2       05
1       2       5
Time taken: 2.119 seconds, Fetched 2 row(s)

spark-sql> show partitions test_90;
dt=05
dt=5
Time taken: 0.161 seconds, Fetched 2 row(s)

spark-sql> select * from test_90 where dt='05';
1       2       05
Time taken: 0.252 seconds, Fetched 1 row(s)

spark-sql> explain insert into table test_90 partition (dt=05) values("1","2");
plan
== Physical Plan ==
Execute InsertIntoHiveTable `db1`.`test_90`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [dt=Some(5)], false, false, [a, b]
+- LocalTableScan [a#109, b#110]{code}
This will cause problems in reading data after the user switches to spark3. The root cause is that in the process of partition field resolution, Spark3 has a process of strongly converting this string type, which will cause partition `05` to lose the previous `0`

So I think we have two solutions:

one is to record the risk clearly in the migration document, and the other is to repair this case, because we internally keep the partition of string type as string type, regardless of whether single or double quotation marks are added.
Bug: Filtering by row_index always returns empty results
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 05/Jan/23 09:17
Affects Version/s: 3.4.0

Queries that include a filter with row_index currently always return an empty result. This is because we consider all metadata attributes constant per file [here|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L76] and the filter then always evaluates to false.

This should be fixed as a follow up to SPARK-41791
Bug: Use approx_percentile function running slower than percentile in spark3 
Status: Open | Priority: Major | Resolution: 
Created: 29/Jan/23 02:13
Affects Version/s: 3.2.1

approx_percentile(end_ts-start_ts,0.9) cost_p90

in spark3 , it use objectHashAggregate method , but it shuffle very slow. when i use percentile , it become fast. i dont know the reson, i think approx_percentile should fast.
Bug: groupByKey creates columns that may conflict with exising columns
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jan/23 14:43
Affects Version/s: 3.0.3

Calling {{ds.groupByKey(func: V => K)}} creates columns to store the key value. These columns may conflict with columns that already exist in {{ds}}. Function {{Dataset.groupByKey.agg}} accounts for this with a very specific rule, which has some surprising weaknesses:
{code:scala}
spark.range(1)
  // groupByKey adds column 'value'
  .groupByKey(id => id)
  // which cannot be referenced, though it is suggested
  .agg(count("value"))
{code}
{code:java}
org.apache.spark.sql.AnalysisException: Column 'value' does not exist. Did you mean one of the following? [value, id];
{code}
An existing 'value' column can be referenced:
{code:scala}
// dataset with column 'value'
spark.range(1).select($"id".as("value")).as[Long]
  // groupByKey adds another column 'value'
  .groupByKey(id => id)
  // agg accounts for the extra column and excludes it when resolving 'value'
  .agg(count("value"))
  .show()
{code}
{code:java}
+---+------------+
|key|count(value)|
+---+------------+
|  0|           1|
+---+------------+
{code}
While column suggestion shows both 'value' columns:
{code:scala}
spark.range(1).select($"id".as("value")).as[Long]
  .groupByKey(id => id)
  .agg(count("unknown"))
{code}
{code:java}
org.apache.spark.sql.AnalysisException: Column 'unknown' does not exist. Did you mean one of the following? [value, value]
{code}

However, {{mapValues}} introduces another 'value' column, which should be referencable, but it breaks the exclusion introduced by {{agg}}:
{code:scala}
spark.range(1)
  // groupByKey adds column 'value'
  .groupByKey(id => id)
  // adds another 'value' column
  .mapValues(value => value)
  // which cannot be referenced in agg
  .agg(count("value"))
{code}
{code:java}
org.apache.spark.sql.AnalysisException: Reference 'value' is ambiguous, could be: value, value.
{code}
Bug: Support overriding the driver/executors container name
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 14/Jan/23 00:20
Affects Version/s: 3.4.0

When we don't provide a pod template for driver/executor pods, or we provide a pod template without the container name, spark uses {{spark-kubernetes-driver}} as container name for the driver and {{spark-kubernetes-executor}} as container name for the executors pods.

I suggest adding two new config {{spark.kubernetes.driver.container.name}} and {{spark.kubernetes.executor.container.name}} to override the default containers names.
Bug: Remove redundant logging of TaskMetrics internal accumulators in JsonProtocol event logs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/23 00:25
Affects Version/s: 3.0.0

Spark's JsonProtocol event logs (used by the history server) contain redundancy in how TaskMetrics are represented in SparkListenerTaskEnd events:
 * The "Task Metrics" field is a map from metric names to values.
 * Under the hood, each metric is implemented using an accumulator and those accumulator values are redundantly stored in the `Task Info`.`Accumulables` field. These Accumulable entries contain the metric value from the task, plus the cumulative "sum-so-far" from the completed tasks in that stage.

The Spark History Server doesn't rely on the redundant information in the Accumulables field.

I believe that this redundancy was introduced back in SPARK-10620 when Spark 1.x's separate TaskMetrics implementation was replaced by the current accumulator-based version.

I think that we should eliminate this redundancy by skipping JsonProtocol logging of the TaskMetric accumulators. Although I think it's somewhat unlikely that third-party code is relying on the presence of that redundant information, I think we should hedge by adding an internal configuration flag to re-enable the redundant logging if needed.
Bug: Implement SparkSession.sql's string formatter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 07:24
Affects Version/s: 4.0.0

{code}
**********************************************************************
File "/.../spark/python/pyspark/sql/connect/session.py", line 345, in pyspark.sql.connect.session.SparkSession.sql
Failed example:
    spark.sql(
        "SELECT * FROM range(10) WHERE id > {bound1} AND id < {bound2}", bound1=7, bound2=9
    ).show()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.session.SparkSession.sql[1]>", line 1, in <module>
        spark.sql(
    TypeError: sql() got an unexpected keyword argument 'bound1'
**********************************************************************
File "/.../spark/python/pyspark/sql/connect/session.py", line 355, in pyspark.sql.connect.session.SparkSession.sql
Failed example:
    spark.sql(
        "SELECT {col} FROM {mydf} WHERE id IN {x}",
        col=mydf.id, mydf=mydf, x=tuple(range(4))).show()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.session.SparkSession.sql[3]>", line 1, in <module>
        spark.sql(
    TypeError: sql() got an unexpected keyword argument 'col'
{code}
Bug: Sorting issue with partitioned-writing and planned write optimization disabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 21:05
Affects Version/s: 3.4.0

Spark 3.4.0 introduced option {{{}spark.sql.optimizer.plannedWrite.enabled{}}}, which is enabled by default. When disabled, partitioned writing loses in-partition order when spilling occurs.

This is related to SPARK-40885 where setting option {{spark.sql.optimizer.plannedWrite.enabled}} to {{true}} will remove the existing sort (for {{day}} and {{{}id{}}}) entirely.

Run this with 512m memory and one executor, e.g.:
{code}
spark-shell --driver-memory 512m --master "local[1]"
{code}
{code:scala}
import org.apache.spark.sql.SaveMode

spark.conf.set("spark.sql.optimizer.plannedWrite.enabled", false)

val ids = 2000000
val days = 2
val parts = 2

val ds = spark.range(0, days, 1, parts).withColumnRenamed("id", "day").join(spark.range(0, ids, 1, parts))

ds.repartition($"day")
  .sortWithinPartitions($"day", $"id")
  .write
  .partitionBy("day")
  .mode(SaveMode.Overwrite)
  .csv("interleaved.csv")
{code}
Check the written files are sorted (states OK when file is sorted):
{code:bash}
for file in interleaved.csv/day\=*/part-*
do
  echo "$(sort -n "$file" | md5sum | cut -d " " -f 1)  $file"
done | md5sum -c
{code}
Files should look like this
{code}
0
1
2
...
1048576
1048577
1048578
...
{code}
But they look like
{code}
0
1048576
1
1048577
2
1048578
...
{code}
The cause issue is the same as in SPARK-40588. A sort (for {{{}day{}}}) is added on top of the existing sort (for {{day}} and {{{}id{}}}). Spilling interleaves the sorted spill files.

{code}
Sort [input[0, bigint, false] ASC NULLS FIRST], false, 0
+- AdaptiveSparkPlan isFinalPlan=false
   +- Sort [day#2L ASC NULLS FIRST, id#4L ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(day#2L, 200), REPARTITION_BY_COL, [plan_id=30]
         +- BroadcastNestedLoopJoin BuildLeft, Inner
            :- BroadcastExchange IdentityBroadcastMode, [plan_id=28]
            :  +- Project [id#0L AS day#2L]
            :     +- Range (0, 2, step=1, splits=2)
            +- Range (0, 2000000, step=1, splits=2)
{code}
Bug: JsonProtocol should skip logging of push-based shuffle read metrics when push-based shuffle is disabled
Status: Open | Priority: Major | Resolution: 
Created: 26/Jan/23 21:24
Affects Version/s: 3.4.0

This is a followup to SPARK-36620:

When push-based shuffle is disabled (the default), I think that we should skip the logging of the new push-based shuffle read metrics. Because these metrics are logged for every task, they will add significant additional size to Spark event logs. It would be great to avoid this cost in cases where it's not necessary.
Bug: Support StreamingQueryListener for DataFrame.observe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 02:50
Affects Version/s: 3.4.0


{code:java}
**********************************************************************
1334
File "/__w/spark/spark/python/pyspark/sql/connect/dataframe.py", line 619, in pyspark.sql.connect.dataframe.DataFrame.observe
1335
Failed example:
1336
    observation.get
1337
Exception raised:
1338
    Traceback (most recent call last):
1339
      File "/usr/lib/python3.9/doctest.py", line 1336, in __run
1340
        exec(compile(example.source, filename, "single",
1341
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.observe[6]>", line 1, in <module>
1342
        observation.get
1343
      File "/__w/spark/spark/python/pyspark/sql/utils.py", line 378, in wrapped
1344
        raise NotImplementedError()
1345
    NotImplementedError
1346
**********************************************************************
1347
File "/__w/spark/spark/python/pyspark/sql/connect/dataframe.py", line 642, in pyspark.sql.connect.dataframe.DataFrame.observe
1348
Failed example:
1349
    spark.streams.addListener(MyErrorListener())
1350
Exception raised:
1351
    Traceback (most recent call last):
1352
      File "/usr/lib/python3.9/doctest.py", line 1336, in __run
1353
        exec(compile(example.source, filename, "single",
1354
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.observe[9]>", line 1, in <module>
1355
        spark.streams.addListener(MyErrorListener())
1356
    AttributeError: 'SparkSession' object has no attribute 'streams'
1357
**********************************************************************
{code}
Bug: Guides for implementing custom data sources & sinks for structured streaming
Status: Open | Priority: Major | Resolution: 
Created: 25/Jan/23 16:32
Affects Version/s: 3.3.1

Please create programming guides about how to implement custom data sources and sinks for structured streaming (or include them in the "Structured Streaming" guide). The data source part has been [asked on SOF|https://stackoverflow.com/questions/47604184] but the accepted answer is not comprehensive and a bit dated (referring to old API versions).
Bug: QueryExecutionListener and Observation API, df.observe do not work with `foreach` action.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 17:28
Affects Version/s: 3.1.3

Observation API, {{observe}} dataframe transformation, and custom QueryExecutionListener.
Do not work with {{foreach}} or {{foreachPartition actions.}}
{{This is due to }}QueryExecutionListener functions do not trigger on queries whose action is {{foreach}} or {{{}foreachPartition{}}}.
But the Spark GUI SQL tab sees this query as SQL query and shows its query plans and etc.


here is the code to reproduce it:
https://gist.github.com/GrigorievNick/e7cf9ec5584b417d9719e2812722e6d3
Bug: DeduplicateRelations rule breaks plan when co-grouping the same DataFrame
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/23 10:53
Affects Version/s: 3.0.3

Co-grouping two DataFrames that share references breaks on the DeduplicateRelations rule:
{code:java}
val df = spark.range(3)

val left_grouped_df = df.groupBy("id").as[Long, Long]
val right_grouped_df = df.groupBy("id").as[Long, Long]

val cogroup_df = left_grouped_df.cogroup(right_grouped_df) {
  case (key, left, right) => left
}

cogroup_df.explain()
{code}
{code:java}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- SerializeFromObject [input[0, bigint, false] AS value#12L]
   +- CoGroup, id#0: bigint, id#0: bigint, id#0: bigint, [id#13L], [id#13L], [id#13L], [id#13L], obj#11: bigint
      :- !Sort [id#13L ASC NULLS FIRST], false, 0
      :  +- !Exchange hashpartitioning(id#13L, 200), ENSURE_REQUIREMENTS, [plan_id=16]
      :     +- Range (0, 3, step=1, splits=16)
      +- Sort [id#13L ASC NULLS FIRST], false, 0
         +- Exchange hashpartitioning(id#13L, 200), ENSURE_REQUIREMENTS, [plan_id=17]
            +- Range (0, 3, step=1, splits=16)
{code}

The DataFrame cannot be computed:
{code:java}
cogroup_df.show()
{code}
{code:java}
java.lang.IllegalStateException: Couldn't find id#13L in [id#0L]
{code}

The rule replaces `id#0L` on the right side with `id#13L` while replacing all occurrences in `CoGroup`. Some occurrences of `id#0L` in `CoGroup`refer to the left side and should not be replaced. Further, `id#0L` of the right deserializer is not replaced.
Bug: Remove logging of Accumulables in Task/Stage start events in JsonProtocol
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/23 00:34
Affects Version/s: 3.0.0

Spark's JsonProtocol event logs (used by the history server) are impacted by a race condition when tasks / stages finish very quickly:

The SparkListenerTaskStart and SparkListenerStageSubmitted events contain mutable TaskInfo and StageInfo objects, which in turn contain Accumulables fields. When a task or stage is submitted, Accumulables is initially empty. When the task or stage finishes, this field is updated with values from the task.

If a task or stage finishes before the start event has been logged by the event logging listener then the _start_ event will contain the Accumulable values from the task or stage _end_ event. 

This information isn't used by the History Server and contributes to wasteful bloat in event log sizes. In one real-world log, I found that ~10% of the uncompressed log size was due to these redundant Accumulable fields.

I propose that we update JsonProtocol to skip the logging of this field for Start/Submitted events.