av9ash/GitBugs huggingface dataset:
Bug: The file size of `sql/hive/target/unit-tests.log` is too big
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Sep/22 03:51
Affects Version/s: 3.4.0

SPARK-6908  set  the file appender log level threshold of the hive UTs from info to debug
,but didn't explain why.

 

When I run 
{code:java}
mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive {code}
the size of the whole Spark directory is about 22G, and the size of `sql/hive/target/unit-tests.log` is 12G. However, the debug level logs in the log files seem worthless, but it takes up a lot of disk space.

 
{code:java}
# Set the logger level of File Appender to WARN
log4j.appender.FA.Threshold = DEBUG {code}
And the original comment of this config is `{{{}Set the logger level of File Appender to WARN{}}}`,  but the {{warn}} level has not been used
Bug: Support `NullType` in `ColumnarBatchRow`
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 17/Sep/22 00:17
Affects Version/s: 3.4.0

`ColumnarBatchRow.get()` does not support `NullType` currently. Support `NullType` in `ColumnarBatchRow` so that `NullType` can be partition column type.
Bug: Improve the error message if the DSv2 source is disabled but DSv1 streaming source is not available
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Sep/22 03:04
Affects Version/s: 3.4.0

If the V2 data source is disabled, current behavior will fallback to use V1 data source. But it will throw error when the DSv1 is not available. Update the error message to indicate what config variable (spark.sql.streaming.disabledV2MicroBatchReaders) needs to be modified in order to enable the V2 data source.
Bug: Extend the partitioning options of the JDBC data source
Status: In Progress | Priority: Minor | Resolution: 
Created: 19/Sep/22 08:00
Affects Version/s: 3.3.0

This proposes to extend the available partitioning options for the JDBC data source.

Partitioning options allow to read data using multiple workers connected to the target RDBMS. This can improve the performance of data extraction, under the right circumstances.

Currently the only available partitioning and parallelization option for reading from databases is to specify lowerBound, upperBound, together with numPartitions and partitionColumn. The Spark JDBC data source will then use multiple partitions, and thus workers, to read from the RDBMS.

This proposes to add a similar, however complementary, mechanism for partitioning, where a user-provided list of values is used to compute the target partitions.

This provides a way to split the data extraction work among workers that could be aligned with the database physical (partitioned and/or indexed) structure, as in the following example:
{code:java}
option("partitionColumn", "region").
option("numPartitions", 3).
option("partitionColValues", "'eastern', 'central', 'western'").  {code}
This feature is motivated for performance reasons, to scale and speed up data extraction from:

 - list partitioned tables, available in Oracle and PostgreSQL

 - this is also applicable to tables stored in B*Tree indexes, such as in Oracle's IOTs (Index Organized Tables) and SQL Server's Clustered Indexes.
Bug: Fix the wrong description related to `spark.shuffle.service.db` in the document
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Sep/22 02:52
Affects Version/s: 3.4.0

From the context from pr of SPARK-17321, YarnShuffleService will persist data into Level/RocksDB when Yarn NM recovery is enabled.  This behavior is not controlled by `spark.shuffle.service.db.enabled` and is not always enabled.  

So the description of `spark.shuffle.service.db.enabled` in `spark-standalone.md` is misleading
Bug: Upgrade RoaringBitmap to 0.9.32
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Sep/22 07:50
Affects Version/s: 3.4.0

[https://github.com/RoaringBitmap/RoaringBitmap/compare/0.9.31...0.9.32]

Two bug fix:
 * [#574|https://github.com/RoaringBitmap/RoaringBitmap/issues/574] [Bug in RoaringBatchIterator.clone() (|https://github.com/RoaringBitmap/RoaringBitmap/commit/6a57f8cdcef3694a0724f597e36534f31e27d8ac] [#575|https://github.com/RoaringBitmap/RoaringBitmap/pull/575] [)|https://github.com/RoaringBitmap/RoaringBitmap/commit/6a57f8cdcef3694a0724f597e36534f31e27d8ac]
 * [Roaring64Bitmap.forInRange correctness issues (|https://github.com/RoaringBitmap/RoaringBitmap/commit/d53aeb5af83e15d8a841d9448c3ecc60d2402f76] [#578|https://github.com/RoaringBitmap/RoaringBitmap/pull/578] [)|https://github.com/RoaringBitmap/RoaringBitmap/commit/d53aeb5af83e15d8a841d9448c3ecc60d2402f76]
Bug: Split FlatMapGroupsWithState down to multiple test suites
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Sep/22 03:11
Affects Version/s: 3.4.0

The suite now exceeds 1800+ lines which is too huge to maintain further.

After quick glance, it looks like the suite can be broken down to three parts (suites) as each part doesn't couple with other.

1. GroupState tests
2. E2E tests which don't test initial state
3. E2E tests which test initial state
Bug: Wrong version of okio in spark-deps file
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 13/Sep/22 15:11
Affects Version/s: 3.4.0

kubernetes-client cascade depends on okio 1.15.0 , which is a compile scope dependency for spark

selenium-java depends on okio 1.14.0, which is a test scope dependency for spark

 

but Spark Project Assembly  choice okio 1.14.0 as compile scope dependency and the version in spark-deps file also 1.14.0.
Bug: The default logging should go to stderr
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Sep/22 03:20
Affects Version/s: 3.4.0

https://github.com/apache/spark/pull/34895?notification_referrer_id=NT_kwDOABaC6bIyODE5MzIzNjAwOjE0NzUzMDU#discussion_r967837334
Bug: Unify `initDB` method in `DBProvider`
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Sep/22 02:56
Affects Version/s: 3.4.0

There are 2 `initDB` in `DBProvider`
 # DB initDB(DBBackend dbBackend, File dbFile, StoreVersion version, ObjectMapper mapper)
 # DB initDB(DBBackend dbBackend, File file)

and the second one only used by test ShuffleTestAccessor, we can explore whether can keep only the first method
Bug: CTE outer reference nested in CTE main body cannot be resolved
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Sep/22 02:38
Affects Version/s: 3.3.0

AnalysisException "Table or view not found" is thrown when a CTE reference occurs in an inner CTE definition nested in the outer CTE's main body FROM clause. E.g.,

{code}
WITH cte_outer AS (
  SELECT 1
)
SELECT * FROM (
  WITH cte_inner AS (
    SELECT * FROM cte_outer
  )
  SELECT * FROM cte_inner
)
{code}
Bug: upgrade com.fasterxml.jackson.dataformat:jackson-dataformat-yaml from 2.13.3 to 2.13.4
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Sep/22 15:04
Affects Version/s: 3.4.0

[CVE-2022-25857|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-25857]

[SNYK-JAVA-ORGYAML|https://security.snyk.io/vuln/SNYK-JAVA-ORGYAML-2806360]
Bug: str_to_map should accept non-foldable delimiter arguments
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Sep/22 21:55
Affects Version/s: 3.4.0

Currently, str_to_map requires the delimiter arguments to be foldable expressions. For example, the following doesn't work in Spark SQL:
{noformat}
drop table if exists maptbl;
create table maptbl as select ',' as del1, ':' as del2, 'a:1,b:2,c:3' as str;
insert into table maptbl select '%' as del1, '-' as del2, 'a-1%b-2%c-3' as str;
select str, str_to_map(str, del1, del2) from maptbl;
{noformat}
You get the following error:
{noformat}
str_to_map's delimiters must be foldable.; line 1 pos 12;
{noformat}
However, the above example SQL statements do work in Hive 2.3.9. There, you get:
{noformat}
+--------------+----------------------------+
|     str      |            _c1             |
+--------------+----------------------------+
| a:1,b:2,c:3  | {"a":"1","b":"2","c":"3"}  |
| a-1%b-2%c-3  | {"a":"1","b":"2","c":"3"}  |
+--------------+----------------------------+
2 rows selected (0.13 seconds)
{noformat}
It's unlikely that an input table would have the needed delimiters in columns. The use-case is more likely to be something like this, where the delimiters are determined based on some other value:
{noformat}
select
  str,
  str_to_map(str, ',', if(region = 0, ':', '#')) as m
from
  maptbl2;
{noformat}
Bug: `np.bool` will be deprecated
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Sep/22 11:21
Affects Version/s: 3.3.0

Using `np.bool` generates this warning: 
{quote}
UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.
3070E                     `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
3071E                   Deprecated in NumPy 1.20; for more details and guidance: [https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations]


{quote}
 

See Numpy's deprecation statement here: [https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations]
Bug: Remove too old TODO for JdbcRDD
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 20/Sep/22 03:04
Affects Version/s: 3.4.0

According to the legacy document of JdbcRDD, we need to expose a jdbcRDD function in SparkContext.

In fact, this is a very old TODO and we need to revisit if this is still necessary. Since Spark SQL is the new core, I'm not sure if anyone is interested in a new API to create jdbc RDD.
Bug: Deduplication of WorkerOffer build in CoarseGrainedSchedulerBackend
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 30/Sep/22 07:07
Affects Version/s: 3.4.0

WorkerOffer build in CoarseGrainedSchedulerBackend is repeated two different places with exact same parameters. We can deduplicate and increase readability by moving that to a private function
Bug: Spark Streaming metrics name don't need application name
Status: In Progress | Priority: Trivial | Resolution: 
Created: 20/Sep/22 12:15
Affects Version/s: 3.2.2

Spark  StreamingSource  Metrics sourceName is inappropriate.The label now looks like `application_xxxxx_xxxx_driver_NetworkWordCount_StreamingMetrics_streaming_lastCompletedBatch_processingEndTime `, instead of `application_xxxxx_xxxx_driver_StreamingMetrics_streaming_lastCompletedBatch_processingEndTime`, the Spark app name is not need.

This makes it hard to use metrics for different Spark applications over time. And this makes the metrics sourceName standard inconsistent.
{code:java}
//代码占位符

private[streaming] class StreamingSource(ssc: StreamingContext) extends Source {
  override val metricRegistry = new MetricRegistry
  override val sourceName = "%s.StreamingMetrics".format(ssc.sparkContext.appName)
....
}{code}
And for example, other metrics sourceName don't have appName.
{code:java}
//代码占位符
private[spark] class LiveListenerBusMetrics(conf: SparkConf)
  extends Source with Logging {

  override val sourceName: String = "LiveListenerBus"
  override val metricRegistry: MetricRegistry = new MetricRegistry
...
}

{code}
Bug: Review and fix issues in Scala/Java API docs
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 09/Aug/21 05:39
Affects Version/s: 3.2.0

Compare the 3.2.0 API doc with the latest release version 3.1.2. Fix the following issues:

* Add missing `Since` annotation for new APIs
* Remove the leaking class/object in API doc
Bug: Downgrade scala-maven-plugin to 4.3.0
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Aug/21 18:20
Affects Version/s: 3.2.0

This is the same issue as SPARK-34007. We need to downgrade the version of scala-maven-plugin to 4.3.0, in order to avoid error on building Hadoop 2.7 package
Bug: HDFSBackedStateStore and RocksDBStateStore have bugs on prefix scan
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 31/Aug/21 02:46
Affects Version/s: 3.2.0

In RocksDB state store provider implementation, we leverage iterators on prefix scan, which are being closed in rollback() method.

While this works now for session window since state store instance in read physical plan will always call abort, it could bring correctness issue for stateful operator which doesn't instantiate two different physical plans on read and write.

We should make sure these iterators get closed to let these iterators don't affect multiple micro-batches (plays as side-effect).

In HDFSBackedStateStore, we copy both data map and prefix key map on creating new map for writing, but we copied prefix key map as it is, which does shallow copy on value side (the type of value is Set), which is broken on reloading aborted version.
Bug: Stage has all tasks finished but with ongoing finalization can cause job hang
Status: Resolved | Priority: Blocker | Resolution: Won't Fix
Created: 23/Aug/21 02:54
Affects Version/s: 3.2.0

 

For a stage that all tasks are finished but with ongoing finalization can lead to job hang. The problem is that such stage is considered as a "missing" stage (see [https://github.com/apache/spark/blob/a47ceaf5492040063e31e17570678dc06846c36c/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L719-L721).] And it breaks the original assumption that a "missing" stage must have tasks to run. 

Normally, if stage A is the parent of (result) stage B and all tasks have finished in stage A, stage A will be skipped directly when submitting stage B. However, with this bug, stage A will be submitted. And submitting a stage with no tasks to run would not be able to add its child stage into the waiting stage list, which leads to the job hang in the end.

 

The example to reproduce:

First, change `MyRDD` to allow it to compute:
{code:java}
override def compute(split: Partition, context: TaskContext): Iterator[(Int, Int)] = {
   Iterator.single((1, 1))
 }{code}

 Then run this test:
{code:java}
test("Job hang") {
  initPushBasedShuffleConfs(conf)
  conf.set(config.SHUFFLE_MERGER_LOCATIONS_MIN_STATIC_THRESHOLD, 5)
  DAGSchedulerSuite.clearMergerLocs
  DAGSchedulerSuite.addMergerLocs(Seq("host1", "host2", "host3", "host4", "host5"))
  val latch = new CountDownLatch(1)
  val myDAGScheduler = new MyDAGScheduler(
    sc,
    sc.dagScheduler.taskScheduler,
    sc.listenerBus,
    sc.env.mapOutputTracker.asInstanceOf[MapOutputTrackerMaster],
    sc.env.blockManager.master,
    sc.env) {
    override def scheduleShuffleMergeFinalize(stage: ShuffleMapStage): Unit = {
      // By this, we can mimic a stage with all tasks finished
      // but finalization is incomplete.
      latch.countDown()
    }
  }
  sc.dagScheduler = myDAGScheduler
  sc.taskScheduler.setDAGScheduler(myDAGScheduler)
  val parts = 20
  val shuffleMapRdd = new MyRDD(sc, parts, Nil)
  val shuffleDep = new ShuffleDependency(shuffleMapRdd, new HashPartitioner(parts))
  val reduceRdd1 = new MyRDD(sc, parts, List(shuffleDep), tracker = mapOutputTracker)
  reduceRdd1.countAsync()
  latch.await()
  // scalastyle:off
  println("=========after wait==========")
  // set _shuffleMergedFinalized to true can avoid the hang.
  // shuffleDep._shuffleMergedFinalized = true
  val reduceRdd2 = new MyRDD(sc, parts, List(shuffleDep))
  reduceRdd2.count()
}
{code}
Bug: Prohibit update mode in native support of session window
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 10/Aug/21 03:16
Affects Version/s: 3.2.0

The semantic of update mode for native support of session window seems to be broken.

Strictly saying, it doesn't break the semantic based on our explanation of update mode:

{quote}
Update Mode - Only the rows that were updated in the Result Table since the last trigger will be written to the external storage (available since Spark 2.1.1). Note that this is different from the Complete Mode in that this mode only outputs the rows that have changed since the last trigger. If the query doesn’t contain aggregations, it will be equivalent to Append mode.
{quote}

But given the grouping key is changing due to the nature of session window, there is no way to "upsert" the output into destination. If end users try to "upsert" the output based on the grouping key, it is high likely that a single session window output will be written into multiple rows across multiple updates.
Bug: Review and fix issues in 3.2.0 Documents
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 26/Aug/21 07:34
Affects Version/s: 3.2.0

Compare the 3.2.0 doc with the latest release version 3.1.2. Fix the following issues:

* Add missing `Since` annotation for new APIs
* Remove the leaking class/object in API doc
* Revise SQL doc
Bug: YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor
Status: Open | Priority: Critical | Resolution: 
Created: 06/Aug/21 16:21
Affects Version/s: 2.4.8

When dynamic allocation is enabled, executors that deallocate rely on the shuffle server to hold blocks and supply them to remaining executors.

When YARN Shuffle Server restarts (either intentionally or due to a crash), it loses block information and relies on being able to contact Executors (the locations of which it durably stores) to refetch the list of blocks.

This mutual dependency on the other to hold block information fails fatally under some common scenarios.

For example, if a Spark application is running under dynamic allocation, some amount of executors will almost always shut down.

If, after this has occurred, any shuffle server crashes, or is restarted (either directly when running as a standalone service, or as part of a YARN node manager restart) then there is no way to restore block data and it is permanently lost.

Worse, when Executors try to fetch blocks from the shuffle server, the shuffle server cannot location the exeutor, decides it doesn't exist, treats it as a fatal exception, and causes the application to terminate and crash.

Thus, in a real world scenario that we observe on a 1000+ node multi-tenant cluster  where dynamic allocation is on by default, a rolling restart of the YARN node managers will cause ALL jobs that have deallocated any executor and have shuffles or transferred blocks to the shuffle server in order to shut down, to crash.
Bug: SessionWindowStateStoreSaveExec should not filter input rows against watermark
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 11/Aug/21 12:52
Affects Version/s: 3.2.0

SessionWindowStateStoreSaveExec receives all sessions including existing sessions into input rows and stores as they are. That said, we should not filter out input rows before storing into state store, but we do. 

Fortunately it hasn't showed any actual problem due to the nature how we deal with watermark against micro-batch and it seems hard to come up with the broken case, but it should be better to fix it before someone succeeds to touch the possible edge case.
Bug: Use CAST in parsing of dates/timestamps with default pattern
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/21 18:01
Affects Version/s: 3.3.0

In functions, CSV/JSON datasources and other places, when the pattern is default, use CAST logic in parsing strings to dates/timestamps.

Currently, TimestampFormatter.getFormatter() applies the default pattern *yyyy-MM-dd  HH:mm:ss* when the pattern is not set, see https://github.com/apache/spark/blob/f2492772baf1d00d802e704f84c22a9c410929e9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L344 . Instead of that, need to create a special formatter which invokes the cast logic.
Bug: Support left outer join build left or right outer join build right in shuffled hash join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/21 09:48
Affects Version/s: 3.2.0

Currently spark sql does not support build left side when left outer join (or build right side when right outer join).

However, in our production environment, there are a large number of scenarios where small tables are left join large tables, and many times, large tables have data skew (currently AQE can't handle this kind of skew).

Inspired by SPARK-32399, we can use similar ideas to realize left outer join build left.

I think this treatment is very meaningful, but I don’t know how members consider this matter?
Bug: timestamp type column analyze result is wrong
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Aug/21 07:39
Affects Version/s: 3.1.1

when we create table with timestamp column type, the min and max data of the analyze result for the timestamp column is wrong

eg:

{code}
> select * from a;
{code}

{code}
2021-08-15 15:30:01
Time taken: 2.789 seconds, Fetched 1 row(s)
spark-sql> desc formatted a a;
col_name a
data_type timestamp
comment NULL
min 2021-08-15 07:30:01.000000
max 2021-08-15 07:30:01.000000
num_nulls 0
distinct_count 1
avg_col_len 8
max_col_len 8
histogram NULL
Time taken: 0.278 seconds, Fetched 10 row(s)
spark-sql> desc a;
a timestamp NULL
Time taken: 1.432 seconds, Fetched 1 row(s)
{code}

 

reproduce step:

{code}
create table a(a timestamp);
insert into a select '2021-08-15 15:30:01';
analyze table a compute statistics for columns a;
desc formatted a a;
select * from a;
{code}
Bug: Implement lazy materialization for the vectorized Parquet reader
Status: Open | Priority: Major | Resolution: 
Created: 16/Aug/21 18:17
Affects Version/s: 3.3.0

At the moment the Parquet vectorized reader will eagerly decode all the columns that are in the read schema, before any filter has been applied to them. This is costly. Instead it's better to only materialize these column vectors when the data are actually needed.
Bug: Decouple CPU with IO work in vectorized Parquet reader
Status: Open | Priority: Major | Resolution: 
Created: 16/Aug/21 18:26
Affects Version/s: 3.3.0

Currently it seems the vectorized Parquet reader does almost everything in a sequential manner:
1. read the row group using file system API (perhaps from remote storage like S3)
2. allocate buffers and store those row group bytes into them
3. decompress the data pages
4. in Spark, decode all the read columns one by one
5. read the next row group and repeat from 1.

A lot of improvements can be done to decouple the IO and CPU intensive work. In addition, we could parallelize the row group loading and column decoding, and utilizing all the cores available for a Spark task.
Bug: Implement lazy decoding for the vectorized Parquet reader
Status: Open | Priority: Major | Resolution: 
Created: 16/Aug/21 18:20
Affects Version/s: 3.3.0

Currently Spark first decode (e.g., RLE/bit-packed, PLAIN) into column vector and then operate on the decoded data. However, it may be more efficient to directly operate on encoded data, for instance, performing filter or aggregation on RLE-encoded data, or performing comparison over dictionary-encoded string data. This can also potentially work with encodings in Parquet v2 format, such as DELTA_BYTE_ARRAY.