av9ash/GitBugs huggingface dataset:
Bug: Shuffle happens when Coalesce Buckets should occur
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Apr/23 02:35
Affects Version/s: 3.3.1

h1. What I did

I define the following code:

{{from pyspark.sql import SparkSession}}

{{spark = (}}
{{  SparkSession}}
{{    .builder}}
{{    .appName("Bucketing")}}
{{    .master("local[4]")}}
{{    .config("spark.sql.bucketing.coalesceBucketsInJoin.enabled", True)}}
{{    .config("spark.sql.autoBroadcastJoinThreshold", "-1")}}
{{    .getOrCreate()}}
{{)}}

{{df1 = spark.range(0, 100)}}
{{df2 = spark.range(0, 100, 2)}}

{{df1.write.bucketBy(4, "id").mode("overwrite").saveAsTable("t1")}}
{{df2.write.bucketBy(2, "id").mode("overwrite").saveAsTable("t2")}}

{{t1 = spark.table("t1")}}
{{t2 = spark.table("t2")}}

{{t2.join(t1, "id").explain()}}

h1. What happened

There is an Exchange node in the join plan

h1. What is expected

The plan should not contain any Exchange/Shuffle nodes, because {{t1}}'s number of buckets is 4 and {{t2}}'s number of buckets is 2, and their ratio is 2 which is less than 4 ({{spark.sql.bucketing.coalesceBucketsInJoin.maxBucketRatio}}) and [CoalesceBucketsInJoin|https://github.com/apache/spark/blob/c9878a212958bc54be529ef99f5e5d1ddf513ec8/sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/CoalesceBucketsInJoin.scala] should be applied
Bug: Use `BytesWritable.copyBytes` instead of manual copy in `HiveInspectors`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Apr/23 08:16
Affects Version/s: 3.5.0

// BytesWritable.copyBytes() only available since Hadoop2
              // In order to keep backward-compatible, we have to copy the
              // bytes with old apis
              val bw = x.getPrimitiveWritableObject(data)
              val result = new Array[Byte](bw.getLength())
              System.arraycopy(bw.getBytes(), 0, result, 0, bw.getLength())
              result
Bug: Remove unnecessary assert for UninterruptibleThread in KafkaMicroBatchStream
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Apr/23 03:38
Affects Version/s: 3.3.2

The assert 

 
{code:java}
 assert(Thread.currentThread().isInstanceOf[UninterruptibleThread]) {code}
 

found [https://github.com/apache/spark/blob/master/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L239] 

 

is not needed.  The reason is the following

 
 # This assert was put there due to some issues when the old and deprecated KafkaOffsetReaderConsumer is used.  The default offset reader implementation has been changed to KafkaOffsetReaderAdmin which no longer require it run via UninterruptedThread.
 # Even if the deprecated KafkaOffsetReaderConsumer is used, there are already asserts in that impl to check if it is running via UninterruptedThread e.g. [https://github.com/apache/spark/blob/master/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala#L130] thus the assert in KafkaMicroBatchStream is redundant.
Bug: Array Insert Should Throw On 'Pos' Value 0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Apr/23 11:11
Affects Version/s: 3.3.2

Wanted to start a discussion about the following comment:

[https://github.com/apache/spark/pull/38867/files#r1157449697]

If a 'pos' of 0 is provided, the code here operates as if the user provided a 1. This could be confusing to users (two incides now overlap and lead to the same result, which is logically a little strange), but perhaps more compellingly, the other functions relying on a provided index, such as elementAt [here|https://github.com/Daniel-Davies/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L2344], will raise an exception if provided with an index of 0.
Bug: Possible logic issue
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 06/Apr/23 15:53
Affects Version/s: 3.0.0

I found this bug when the following query returned no results:
{code:java}
WHERE cast(leistungsnummer as long) NOT IN (123)){code}
However so did the following query:
{code:java}
WHERE cast(leistungsnummer as long) IN (123)) {code}
This can be simplified to the following:
{code:java}
WHERE NULL IN (123)
{code}
Returns zero results but so does:
{code:java}
WHERE NULL NOT IN (123) {code}
Logically one would assume `NULL IN (123, 456)` would be false.  However if that is the case then you would think `NULL NOT IN (123, 456)` would be true.  It seems they both equate to false.

I don't know if this is a quirk of the SQL spec or if this is a bug in Spark SQL implementation but it is causing unexpected results.
Bug: Support bin pack task scheduling on executors 
Status: Open | Priority: Minor | Resolution: 
Created: 10/Apr/23 22:43
Affects Version/s: 3.3.2

Dynamic allocation only remove or decommission an idle executor. The default task scheduling use round robin to do task assignment on executors. 

For example, we have 4 tasks to run, 4 executors(each has 4 cpu cores). Default task scheduling will assign 1 task per executors. With bin pack, one executor could assign 4 tasks, then dynamic allocation could remove other 3 executors to reduce resource waste.
Bug: Name based access of accumulators from tasks
Status: Open | Priority: Minor | Resolution: 
Created: 03/Apr/23 11:04
Affects Version/s: 3.5.0

At present, accumulators must be registered on the driver and subsequently accessed (i.e., added/reset) by the code running on the executor using the same object. As a result, the accumulator object must be passed throughout the code, which leads to verbosity, particularly in multi-module Spark applications where user code is distributed across multiple modules.

Instead why not have name based access to accumulator so that objects don't need to be passed explicitly? (Sort of global named accumulators)

Currently,
{code:java}
val accName = "custom_acc_experimental" 
val la = new LongAccumulator()
sc.register(la, accName)

sc.parallelize(0 to 10).mapPartitions(partition => {
  la.add(100L)
  partition
}).count{code}
Can be,
{code:java}
val accName = "custom_acc_experimental"
val la = new LongAccumulator()
sc.register(la, accName) 

sc.parallelize(0 to 10).mapPartitions(partition => {
  // Assuming new end-point
  AccumulatorV2.add(accName, 100L)   
  partition 
}).count {code}
I was able to do a [working POC of this|https://github.com/apache/spark/compare/master...mayurdb:spark:experiment] where the user defined accumulators are internally passed to task executable.

Probably a new endpoint needs to be added for users to create such accumulators as handling all current accumulators in this manner would cause pressure on the task flow RPC. I can write a small doc on approach but wanted to get a ack if this seems workable.
Bug: self.deserialized == self.deserialized typo in StorageLevel __eq__()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Apr/23 03:16
Affects Version/s: 3.4.0

We should fix {{self.deserialized == self.deserialized}} with {{self.deserialized == other.deserialized}}

The original expression is always True, which is likely to be a typo.
Bug: `v is v >= 0` typo in pyspark/pandas/config.py
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Apr/23 03:11
Affects Version/s: 3.2.0

By comparing compute.isin_limit and plotting.max_rows, {{v is v}} is likely to be a typo.

We should fix {{v is v >= 0}} with {{{}v >= 0{}}}.
Bug: vendor==vendor typo in ResourceRequest.equals()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Apr/23 03:03
Affects Version/s: 3.0.0

vendor == vendor is always true, this is likely to be a typo.

We should fix `vendor == vendor` with `that.vendor == vendor`, and `discoveryScript == discoveryScript` with `that.discoveryScript == discoveryScript`.
Bug: `df.show` handle null should print NULL instead of null
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Apr/23 07:45
Affects Version/s: 3.4.0

`df.show` handle null should print NULL instead of null to consistent behavior;
{code:java}
Like as the following behavior is currently inconsistent:
``` shell
scala> spark.sql("select decode(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle') as result").show(false)
+------+
|result|
+------+
|null  |
+------+
```
``` shell
spark-sql> DESC FUNCTION EXTENDED decode;
function_desc
Function: decode
Class: org.apache.spark.sql.catalyst.expressions.Decode
Usage:
    decode(bin, charset) - Decodes the first argument using the second argument character set.

    decode(expr, search, result [, search, result ] ... [, default]) - Compares expr
      to each search value in order. If expr is equal to a search value, decode returns
      the corresponding result. If no match is found, then it returns default. If default
      is omitted, it returns null.

Extended Usage:
    Examples:
      > SELECT decode(encode('abc', 'utf-8'), 'utf-8');
       abc
      > SELECT decode(2, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
       San Francisco
      > SELECT decode(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle', 'Non domestic');
       Non domestic
      > SELECT decode(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle');
       NULL

    Since: 3.2.0

Time taken: 0.074 seconds, Fetched 4 row(s)
```
``` shell
spark-sql> select decode(6, 1, 'Southlake', 2, 'San Francisco', 3, 'New Jersey', 4, 'Seattle');
NULL
{code}
Bug: Before batch reading from Kafka, log topic partition, offset range, etc, for debuggin
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Apr/23 21:28
Affects Version/s: 3.4.0

When debugging some slowness issue in structured streaming, it is hard to map a Kafka topic and partition to a Kafka task. Adding some logging in executor might help make it easier.
Bug: Sort entries in the pyspark API reference docs alphabetically
Status: Open | Priority: Trivial | Resolution: 
Created: 22/Apr/23 11:26
Affects Version/s: 3.2.4

Currently the methods on a lot of api reference docs are nor sorted which makes navigation quite hard and unintuitive, e.g. [https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/series.html#reindexing-selection-label-manipulation]

has the following order:
|[{{Series.equals}}|https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.equals.html#pyspark.pandas.Series.equals](other)|Compare if the current value is equal to the other.|
|[{{Series.add_prefix}}|https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.add_prefix.html#pyspark.pandas.Series.add_prefix](prefix)|Prefix labels with string ??prefix??.|
|[{{Series.add_suffix}}|https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.add_suffix.html#pyspark.pandas.Series.add_suffix](suffix)|Suffix labels with string suffix.|
|[{{Series.first}}|https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.first.html#pyspark.pandas.Series.first](offset)|Select first periods of time series data based on a date offset.|
|[{{Series.head}}|https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.head.html#pyspark.pandas.Series.head]([n])|

 

Ideally the methods should be sorted alphabetically to make navigating the docs predictable.
Bug: Bug in INSERT INTO documentation
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 14/Apr/23 09:00
Affects Version/s: 3.3.2

I think there is a bug in this page [https://spark.apache.org/docs/3.1.2/sql-ref-syntax-dml-insert-into.html]

The following SQL statement does not look valid based on the contents of the "applicants" table.


{code:java}
INSERT INTO students
     FROM applicants SELECT name, address, id applicants WHERE qualified = true; {code}

Specifically, "id applicants" should possibly be changed to "student_id"
Bug: Comment of DAGScheduler.submitMissingTasks is wrong
Status: Closed | Priority: Trivial | Resolution: Invalid
Created: 11/Apr/23 13:49
Affects Version/s: 3.3.2

Comment "Called when stage's parents are available and we can now do its task" of DAGScheduler.submitMissingTasks is wrong.

This method is called when stage's parents are unavailable.

I'd like to fix this.
Bug: Modify Yarn Client's application report to reduce noise
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 03/Apr/23 01:25
Affects Version/s: 3.5.0

Right now, an application report is generated and loggeds every second, this tends to add a lot of noise especially for long running applications. This bloats up the logs and affects user experience.

Also attached is a screenshot showing the issue.

 

!Screen Shot 2023-03-14 at 8.49.54 AM.png!
Bug: hadoop-aws doesn't work
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 01/Dec/20 08:33
Affects Version/s: 3.1.0

According to [HADOOP-16080](https://issues.apache.org/jira/browse/HADOOP-16080) since Apache Hadoop 3.1.1, `hadoop-aws` doesn't work with `hadoop-client-api`. In other words, the regression is that `dev/make-distribution.sh -Phadoop-cloud ...` doesn't make a complete distribution for cloud support. It fails at write operation like the following.

{code}
$ bin/spark-shell --conf spark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID --conf spark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY
20/11/30 23:01:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context available as 'sc' (master = local[*], app id = local-1606806088715).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.0-SNAPSHOT
      /_/

Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_272)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.parquet("s3a://dongjoon/users.parquet").show
20/11/30 23:01:34 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          null|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+


scala> Seq(1).toDF.write.parquet("s3a://dongjoon/out.parquet")
20/11/30 23:02:14 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2)/ 1]
java.lang.NoSuchMethodError: org.apache.hadoop.util.SemaphoredDelegatingExecutor.<init>(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
{code}
Bug: Invalidate new char-like type in public APIs that result incorrect results
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Dec/20 07:35
Affects Version/s: 3.1.0

1. udf
{code:java}
scala> spark.udf.register("abcd", () => "12345", org.apache.spark.sql.types.VarcharType(2))

scala> spark.sql("select abcd()").show
scala.MatchError: CharType(2) (of class org.apache.spark.sql.types.VarcharType)
  at org.apache.spark.sql.catalyst.encoders.RowEncoder$.externalDataTypeFor(RowEncoder.scala:215)
  at org.apache.spark.sql.catalyst.encoders.RowEncoder$.externalDataTypeForInput(RowEncoder.scala:212)
  at org.apache.spark.sql.catalyst.expressions.objects.ValidateExternalType.<init>(objects.scala:1741)
  at org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:175)
  at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
  at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
  at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
  at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
  at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
  at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
  at org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:171)
  at org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:66)
  at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:768)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:611)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:768)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:606)
  ... 47 elided
{code}

2. spark.createDataframe


{code:java}
scala> spark.createDataFrame(spark.read.text("README.md").rdd, new org.apache.spark.sql.types.StructType().add("c", "char(1)")).show
+--------------------+
|                   c|
+--------------------+
|      # Apache Spark|
|                    |
|Spark is a unifie...|
|high-level APIs i...|
|supports general ...|
|rich set of highe...|
|MLlib for machine...|
|and Structured St...|
|                    |
|<https://spark.ap...|
|                    |
|[![Jenkins Build]...|
|[![AppVeyor Build...|
|[![PySpark Covera...|
|                    |
|                    |
|## Online Documen...|
|                    |
|You can find the ...|
|guide, on the [pr...|
+--------------------+
only showing top 20 rows
{code}


3. reader.schema

```
scala> spark.read.schema("a varchar(2)").text("./README.md").show(100)
+--------------------+
|                   a|
+--------------------+
|      # Apache Spark|
|                    |
|Spark is a unifie...|
|high-level APIs i...|
|supports general ...|
```
4. etc
Bug: Fix Spark Release image
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 29/Dec/20 02:32
Affects Version/s: 3.1.0

The release script seems to be broken. This is a blocker for Apache Spark 3.1.0 release.
{code}
$ cd dev/create-release/spark-rm
$ docker build -t spark-rm .
...
exit code: 1
{code}
Bug: Char Varchar values fails w/ match error as partition columns
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 22/Dec/20 09:13
Affects Version/s: 3.1.0


{code:java}
spark-sql> select * from t10 where c0='abcd';
20/12/22 15:43:38 ERROR SparkSQLDriver: Failed in [select * from t10 where c0='abcd']
scala.MatchError: CharType(10) (of class org.apache.spark.sql.types.CharType)
	at org.apache.spark.sql.catalyst.expressions.CastBase.cast(Cast.scala:815)
	at org.apache.spark.sql.catalyst.expressions.CastBase.cast$lzycompute(Cast.scala:842)
	at org.apache.spark.sql.catalyst.expressions.CastBase.cast(Cast.scala:842)
	at org.apache.spark.sql.catalyst.expressions.CastBase.nullSafeEval(Cast.scala:844)
	at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:476)
	at org.apache.spark.sql.catalyst.catalog.CatalogTablePartition.$anonfun$toRow$2(interface.scala:164)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at org.apache.spark.sql.types.StructType.foreach(StructType.scala:102)
	at scala.collection.TraversableLike.map(TraversableLike.scala:238)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
	at org.apache.spark.sql.types.StructType.map(StructType.scala:102)
	at org.apache.spark.sql.catalyst.catalog.CatalogTablePartition.toRow(interface.scala:158)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.$anonfun$prunePartitionsByFilter$3(ExternalCatalogUtils.scala:157)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.$anonfun$prunePartitionsByFilter$3$adapted(ExternalCatalogUtils.scala:156)
{code}


c0 is a partition column, it fails in partition pruning rule
Bug: Document the current behavior for DataStreamWriter.toTable API
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 04/Dec/20 07:36
Affects Version/s: 3.1.0

Follow up work for SPARK-33577 and need to be done before the 3.1 release. As we didn't have full support for the V2 table created in the API, the following documentation work is needed:
 * figure out the effects when configurations are (provider/partitionBy) conflicting with existing table, and document in javadoc of {{toTable}}. I think you'll need to make a matrix and describe which takes effect (table vs input) - creating table vs table exists, DSv1 vs DSv2 (4 different situations should be all documented).
 * document the lack of functionality on creating v2 table in javadoc of {{toTable}}, and guide that they should ensure table is created in prior to avoid the behavior unintended/insufficient table is being created.
Bug: Rename dataSourceRewriteRules and customDataSourceRewriteRules in BaseSessionStateBuilder
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 15/Dec/20 01:59
Affects Version/s: 3.1.0

This is under discussion at https://github.com/apache/spark/pull/30558#discussion_r533885837.

We happened to have rule extension that are not specific to Data source rewrites (SPARK-33612), but we named it so, and people agree with having a good name here.
Bug: Performance regression in Kafka read
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 02/Dec/20 16:54
Affects Version/s: 3.0.0

I have observed a slowdown in the reading of data from kafka on all of our systems when migrating from spark 2.4.5 to Spark 3.0.0 (and Spark 3.0.1)

I have created a sample project to isolate the problem as much as possible, with just a read all data from a kafka topic (see [https://github.com/codegorillauk/spark-kafka-read] ).

With 2.4.5, across multiple runs, 
 I get a stable read rate of 1,120,000 (1.12 mill) rows per second

With 3.0.0 or 3.0.1, across multiple runs,
 I get a stable read rate of 632,000 (0.632 mil) rows per second

The represents a *44% loss in performance*. Which is, a lot.

I have been working though the spark-sql-kafka-0-10 code base, but change for spark 3 have been ongoing for over a year and its difficult to pin point an exact change or reason for the degradation.

I am happy to help fix this problem, but will need some assitance as I am unfamiliar with the spark-sql-kafka-0-10 project.

 

A sample of the data my test reads (note: its not parsing csv - this is just test data)
 1606921800000,001e0610e532,lightsense,tsl250rd,intensity,21853,53.262,acceleration_z,651,ep,290,commit,913,pressure,138,pm1,799,uv_intensity,823,idletime,-372,count,-72,ir_intensity,185,concentration,-61,flags,-532,tx,694.36,ep_heatsink,-556.92,acceleration_x,-221.40,fw,910.53,sample_flow_rate,-959.60,uptime,-515.15,pm10,-768.03,powersupply,214.72,magnetic_field_y,-616.04,alphasense,606.73,AoT_Chicago,053,Racine Ave & 18th St Chicago IL,41.857959,-87.65642700000002,AoT Chicago (S) [C],2017/12/15 00:00:00,
Bug: SQL Adaptive Query Execution QA
Status: Closed | Priority: Critical | Resolution: Done
Created: 17/Dec/20 18:11
Affects Version/s: 3.2.0

Since SPARK-31412 is delivered at 3.0.0, we received and handled many JIRA issues at 3.0.x/3.1.0/3.2.0. This umbrella JIRA issue aims to enable it by default and collect all information in order to do QA for this feature in Apache Spark 3.2.0 timeframe.