av9ash/GitBugs huggingface dataset:
Bug: Make interval multiply and divide's overflow behavior consisitent with other interval operations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/20 18:05
Affects Version/s: 3.0.0

The current behavior of interval multiply and divide follows the ANSI SQL standard when overflow, it is compatible with other interval operations when `spark.sql.ansi.enabled` is true, but not compatible when `spark.sql.ansi.enabled` is false. 

When `spark.sql.ansi.enabled` is false, as the factor is a double value, so we should use java's rounding or truncation behavior for casting double to integrals.
Bug: Forwards-compatibility in JsonProtocol in broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 06:52
Affects Version/s: 3.0.0

JsonProtocol is supposed to provide strong backwards-compatibility and forwards-compatibility guarantees: any version of Spark should be able to read JSON output written by any other version, including newer versions.

However, the forwards-compatibility guarantee is broken for events parsed by "ObjectMapper". If a new field is added to an event parsed by "ObjectMapper" (e.g., https://github.com/apache/spark/commit/6dc5921e66d56885b95c07e56e687f9f6c1eaca7#diff-dc5c7a41fbb7479cef48b67eb41ad254R33), this event cannot be parsed by an old version of Spark History Server.
Bug: Fix java.lang.IndexOutOfBoundsException No group 1 for regexp_extract
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/20 10:23
Affects Version/s: 2.4.5

The current implement of regexp_extract will throws a unprocessed exception show below:

SELECT regexp_extract('1a 2b 14m', '
d+')

 
{code:java}
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 22.0 failed 1 times, most recent failure: Lost task 1.0 in stage 22.0 (TID 33, 192.168.1.6, executor driver): java.lang.IndexOutOfBoundsException: No group 1
[info] at java.util.regex.Matcher.group(Matcher.java:538)
[info] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
[info] at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[info] at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
[info] at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
[info] at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
[info] at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1804)
[info] at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1227)
[info] at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1227)
[info] at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2156)
[info] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
[info] at org.apache.spark.scheduler.Task.run(Task.scala:127)
[info] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
[info] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[info] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
[info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] at java.lang.Thread.run(Thread.java:748)
{code}
 

I think should treat this exception well.
Bug: StringIndexer setOutputCols does not set output cols
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/20 14:36
Affects Version/s: 3.0.0

(Credit to Brooke Wenig for finding it). Quoting:

".. The python code works completely fine, but the scala code is outputting

{code}
strIdx_8278ae6d55b3__output, strIdx_8278ae6d55b3__output, strIdx_8278ae6d55b3__output, strIdx_8278ae6d55b3__output, strIdx_8278ae6d55b3__output, strIdx_8278ae6d55b3__output, strIdx_8278ae6d55b3__output
{code}

for the output of the string indexer, instead of using the column names specified in here:

{code}
val stringIndexer = new StringIndexer()
  .setInputCols(categoricalCols)
  .setOutputCols(indexOutputCols)
  .setHandleInvalid("skip")
{code}

I was expecting the resulting column names to be

{code}
indexOutputCols: Array[String] = Array(host_is_superhostIndex, cancellation_policyIndex, instant_bookableIndex, neighbourhood_cleansedIndex, property_typeIndex, room_typeIndex, bed_typeIndex)
{code}


Indeed I'm pretty sure this is the bug:

{code}
  private def validateAndTransformField(
      schema: StructType,
      inputColName: String,
      outputColName: String): StructField = {
    val inputDataType = schema(inputColName).dataType
    require(inputDataType == StringType || inputDataType.isInstanceOf[NumericType],
      s"The input column $inputColName must be either string type or numeric type, " +
        s"but got $inputDataType.")
    require(schema.fields.forall(_.name != outputColName),
      s"Output column $outputColName already exists.")
    NominalAttribute.defaultAttr.withName($(outputCol)).toStructField()
  }
{code}

The last line does not use the transformed output col name, but the default single output col parameter.
Bug: LIKE returns wrong result from external table using parquet
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/20 06:24
Affects Version/s: 2.0.2

# Write a parquet file with a column in upper case:
{code:scala}
Seq("42").toDF("COL").write.parquet(path)
{code}
# Create an external table on top of the written parquet files with a column in lower case
{code:sql}
CREATE TABLE t1 (col STRING)
USING parquet
OPTIONS (path '$path')
{code}
# Read the table using LIKE
{code:sql}
SELECT * FROM t1 WHERE col LIKE '4%'
{code}
It returns empty set but must be one row with 42.
Bug: Add additional validation into Merge Into
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/20 02:29
Affects Version/s: 3.0.0

Merge Into is currently missing additional validation around:

 1. The lack of any WHEN statements

 2. Single use of UPDATE/DELETE

 3. The first WHEN MATCHED statement needs to have a condition if there are two WHEN MATCHED statements.
Bug: Exclude spark.sql.variable.substitute.depth from removedSQLConfigs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/20 11:15
Affects Version/s: 3.0.0

The spark.sql.variable.substitute.depth SQL config is not used since Spark 2.4 inclusively. By the [https://github.com/apache/spark/pull/27169], the config was placed to SQLConf.removedSQLConfigs. And as a consequence of that when an user set it non-default value (1 for example),  he/she will get an exception. It is acceptable for configs that could impact on the behavior but not for this particular config. Raising of such exception will just make migration to Spark 3.0 more difficult.
Bug: Upgrade to Py4J 0.10.9
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/20 05:41
Affects Version/s: 3.0.0

This issue aims to upgrade Py4J from 0.10.8.1 to 0.10.9.
Py4J 0.10.9 is released with the following fixes.
- https://www.py4j.org/changelog.html#py4j-0-10-9
Bug: CTE that refers to non-existent table with same name causes StackOverflowError
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/20 13:45
Affects Version/s: 2.0.2

The following query causes a StackOverflowError:
{noformat}
WITH t AS (SELECT 1 FROM nonexist.t) SELECT * FROM t
{noformat}

This only happens when the CTE refers to a non-existent table with the same name and a database qualifier. This is caused by a couple of things:
 * {{CTESubstitution}} runs analysis on the CTE, but this does not throw an exception because the table has a database qualifier. The reason is that we don't fail is because we re-attempt to resolve the relation in a later rule.
 * {{CTESubstitution}} replace logic does not check if the table it is replacing has a database, it shouldn't replace the relation if it does. So now we will happily replace {{nonexist.t}} with {{t}}.
 * {{CTESubstitution}} transforms down, this means it will keep replacing {{t}} with itself, creating an infinite recursion.

This is not an issue for master/3.0.
Bug: Revisit the decision of writing parquet TIMESTAMP_MICROS by default
Status: Open | Priority: Major | Resolution: 
Created: 19/Feb/20 13:29
Affects Version/s: 3.0.0

In Spark 3.0, we write out timestamp values as parquet TIMESTAMP_MICROS by default, instead of INT96. This is good in general as Spark can read all kinds of parquet timestamps, but works better with TIMESTAMP_MICROS.

However, this brings some troubles with hive compatibility. Spark can use native parquet writer to write hive parquet tables, which may break hive compatibility if Spark writes TIMESTAMP_MICROS.

We can switch back to INT96 by default, or fix it:
1. when using native parquet writer to write hive parquet tables, write timestamp as INT96.
2. when creating tables in `HiveExternalCatalog.createTable`, don't claim the parquet table is hive compatible if it has timestamp columns.
Bug: Add Columns references should be able to resolve each other
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/20 18:43
Affects Version/s: 3.0.0

In ResolveAlterTableChanges, we have checks that make sure that positional arguments exist and are normalized around case sensitivity for ALTER TABLE ADD COLUMNS. However, we missed the case, where a column in ADD COLUMNS can depend on the position of a column that is just being added.

For example for the schema:
{code:java}
root:
  - a: string
  - b: long
 {code}
 

The following should work:
{code:java}
ALTER TABLE ... ADD COLUMNS (x int AFTER a, y int AFTER x) {code}
Currently, the above statement will throw an error saying that AFTER x cannot be resolved, because x doesn't exist yet.
Bug: Create table like should keep tracksPartitionsInCatalog same with source table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/20 06:02
Affects Version/s: 3.0.0

Table generated by CTL a partitioned table is a partitioned table. But when run ALTER TABLE ADD PARTITION, it will throw AnalysisException: ALTER TABLE ADD PARTITION is not allowed. That's because the default value of {{tracksPartitionsInCatalog}} from CTL always is {{false}}.
Bug: Wrong results of getting time components before 1582 year
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/20 07:58
Affects Version/s: 3.0.0

Here are example that show the issues:
{code:sql}
spark-sql> select hour(timestamp '0010-01-01 00:00:00');
23
spark-sql> select minute(timestamp '0010-01-01 00:00:00');
52
spark-sql> select second(timestamp '0010-01-01 00:00:00');
58
spark-sql> select date_part('milliseconds', timestamp '0010-01-01 00:00:00');
58000.000
spark-sql> select date_part('microseconds', timestamp '0010-01-01 00:00:00');
58000000
{code}
The expected results must be:
* hour = 0
* minute = 0
* second = 0
* milliseconds = 0
* microseconds = 0
Bug: Wrong truncations of timestamps before the epoch to hours and days
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/20 17:24
Affects Version/s: 2.3.0

Truncations to seconds and minutes of timestamps after the epoch are correct:
{code:sql}
spark-sql> select date_trunc('HOUR', '2020-02-11 00:01:02.123'), date_trunc('HOUR', '2020-02-11 00:01:02.789');
2020-02-11 00:00:00	2020-02-11 00:00:00
{code}
but truncations of timestamps before the epoch are incorrect:
{code:sql}
spark-sql> select date_trunc('HOUR', '1960-02-11 00:01:02.123'), date_trunc('HOUR', '1960-02-11 00:01:02.789');
1960-02-11 01:00:00	1960-02-11 01:00:00
{code}
The result must be *1960-02-11 00:00:00 1960-02-11 00:00:00*

The same for the DAY level:
{code:sql}
spark-sql> select date_trunc('DAY', '1960-02-11 00:01:02.123'), date_trunc('DAY', '1960-02-11 00:01:02.789');
1960-02-12 00:00:00	1960-02-12 00:00:00
{code}
The result must be *1960-02-11 00:00:00 1960-02-11 00:00:00*
Bug: Add 'path' field to the 'LoadInstanceEnd' ML listener event
Status: Resolved | Priority: Major | Resolution: Workaround
Created: 17/Feb/20 10:14
Affects Version/s: 3.0.0

The [LoadInstanceEnd|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L92] ML listener event that is added on the scope of SPARK-23674 has no 'path' field which makes it impossible to determine from what path an ML instance was loaded as well as there is no way to get instance's 'uid' via [LoadInstanceStart|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L84] event.

 

The [LoadInstanceEnd|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L92] must be changed to include 'path' field. Please, refer [SaveInstanceStart|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L101] and [SaveInstanceEnd|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L109] events. Both of them have `path` but [LoadInstanceStart|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L84] and [LoadInstanceEnd|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L92] are not.
Bug: Wrong truncations of timestamps before the epoch to minutes and seconds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/20 18:21
Affects Version/s: 2.3.0

Truncations to seconds and minutes of timestamps after the epoch are correct:
{code:sql}
spark-sql> select date_trunc('SECOND', '2020-02-11 00:01:02.123'), date_trunc('SECOND', '2020-02-11 00:01:02.789');
2020-02-11 00:01:02	2020-02-11 00:01:02
{code}
but truncations of timestamps before the epoch are incorrect:
{code:sql}
spark-sql> select date_trunc('SECOND', '1960-02-11 00:01:02.123'), date_trunc('SECOND', '1960-02-11 00:01:02.789');
1960-02-11 00:01:03	1960-02-11 00:01:03
{code}
The result must be *1960-02-11 00:01:02	1960-02-11 00:01:02*

The same for the MINUTE level:
{code:sql}
spark-sql> select date_trunc('MINUTE', '1960-02-11 00:01:01'), date_trunc('MINUTE', '1960-02-11 00:01:50');
1960-02-11 00:02:00	1960-02-11 00:02:00
{code}
The result must be 1960-02-11 00:01:00	1960-02-11 00:01:00
Bug: Dataframe add sameResult and sementicHash method
Status: Resolved | Priority: Major | Resolution: Done
Created: 11/Feb/20 14:05
Affects Version/s: 3.1.0

Sometimes, we want to check whether two dataframes are the same.

There is already an internal API like:
{code:java}
df1.queryExecution.analyzed.sameResult(...)
df1.queryExecution.analyzed.semanticHash(){code}
We can make a public API for this:

Like:
{code:scala}
df1.sameSemantic(df2) // return true if dataframe has the same semantic
df1.semanticHash() // return a semantic hashcode
{code}

CC [~cloud_fan] [~mengxr] [~liangz]
Bug: [Scala] date_format() incorrectly compute time when applied against more than 1 column
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 18/Feb/20 01:56
Affects Version/s: 2.4.4

Here is the sequence of code that demonstrates the weird inconsistent behavior that looks like a bug to me.  

 
{code:java}
scala> spark.conf.set("spark.sql.session.timeZone", "UTC")

scala> case class Event(event_time: String)
defined class Event

scala> val eventDS = Seq(Event("2020-02-18T01:14:14.945Z"), Event("2020-02-18T02:02:02.975Z")).toDS()
eventDS: org.apache.spark.sql.Dataset[Event] = [event_time: string]

scala> eventDS.show(false)
+------------------------+
|event_time              |
+------------------------+
|2020-02-18T01:14:14.945Z|
|2020-02-18T02:02:02.975Z|
+------------------------+

scala> val tsOperationsDF = eventDS.withColumn("add_4_hours", lit($"event_time" + expr("INTERVAL 4 HOURS"))).withColumn("sub_2_hours", lit($"event_time" - expr("INTERVAL 2 HOURS")))
tsOperationsDF: org.apache.spark.sql.DataFrame = [event_time: string, add_4_hours: string ... 1 more field]

scala> tsOperationsDF.show(false)
+------------------------+-----------------------+-----------------------+
|event_time              |add_4_hours            |sub_2_hours            |
+------------------------+-----------------------+-----------------------+
|2020-02-18T01:14:14.945Z|2020-02-18 05:14:14.945|2020-02-17 23:14:14.945|
|2020-02-18T02:02:02.975Z|2020-02-18 06:02:02.975|2020-02-18 00:02:02.975|
+------------------------+-----------------------+-----------------------+

scala> val tsOperationsFormattedDF = eventDS.withColumn("add_4_hours", lit(date_format(($"event_time" + expr("INTERVAL 4 HOURS")),"yyyy-MM-dd hh:mm:ss"))).withColumn("sub_2_hours", lit(date_format(($"event_time" - expr("INTERVAL 2 HOURS")),"yyyy-MM-dd hh:mm:ss")))
tsOperationsFormattedDF: org.apache.spark.sql.DataFrame = [event_time: string, add_4_hours: string ... 1 more field]

scala> tsOperationsFormattedDF.show(false)
+------------------------+-------------------+-------------------+
|event_time              |add_4_hours        |sub_2_hours        |
+------------------------+-------------------+-------------------+
|2020-02-18T01:14:14.945Z|2020-02-18 05:14:14|2020-02-17 11:14:14|
|2020-02-18T02:02:02.975Z|2020-02-18 06:02:02|2020-02-18 12:02:02|
+------------------------+-------------------+-------------------+

{code}
The timestamp operations are identical between the two data frames.  However, the time results are different for the "sub_2_hours" column. The first data frame computed the correct time results, but not the second data frame when date_format() are used.

Any further confirmation and validation of this bug is much appreciated.
Bug: Define LegacyBehaviorPolicy enumeration as the common value for result change configs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/20 09:06
Affects Version/s: 3.0.0

During API auditing for the 3.0 release, we found several new approaches that will change the results silently. For these features, we create a common config value, which is a three-value enumeration. Whenever the new behavior will silently change the results, this config can be reused.
 * Exception(default): Spark will throw an exception while result changes will happen, and remind users to choose between legacy and new behavior.
 * New Behavior: Use the new approach.
 * Legacy: Use the legacy behavior of earlier versions.
Bug: Executors UI shows more active tasks then possible
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/20 16:03
Affects Version/s: 3.0.0

I regularly see the executors web ui showing more active tasks then it has cores. Looking at the code it seems that we track those separately and the message that is sent for task end is asynchronous and thus ends up showing up at the UI much later then the start event.

CoarseGrainedSchedulerBackend on statusUpdate increases the freeCores which then allow scheduler to assign another task, but the taskEndEvent is asynchronous.

We definitely don't want to slow down the scheduling part so not sure how easily it will be to improve.

 

To reproduce I just ran:

val df = sc.makeRDD(1 to 10000000, 6).toDF
val df2 = sc.makeRDD(1 to 10000000, 6).toDF

spark.time(df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" === $"b").write.mode("overwrite").csv("somefile"))

 

And view the executors ui page. I started spark-shell with just 1 core per executor and you  see 2 active tasks
Bug: Wrong truncation of old timestamps to hours and days
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/20 17:28
Affects Version/s: 3.0.0

The date_trunc() function incorrectly truncates old timestamps to HOUR and DAY:
{code:scala}
scala> import  org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> :paste
// Entering paste mode (ctrl-D to finish)

Seq("0010-01-01 01:02:03.123456").toDF()
    .select($"value".cast("timestamp").as("ts"))
    .select(date_trunc("HOUR", $"ts").cast("string"))
    .show(false)

// Exiting paste mode, now interpreting.

+------------------------------------+
|CAST(date_trunc(HOUR, ts) AS STRING)|
+------------------------------------+
|0010-01-01 01:30:17                 |
+------------------------------------+
{code}
the result must be *0010-01-01 01:00:00*
{code:scala}
scala> :paste
// Entering paste mode (ctrl-D to finish)

Seq("0010-01-01 01:02:03.123456").toDF()
    .select($"value".cast("timestamp").as("ts"))
    .select(date_trunc("DAY", $"ts").cast("string"))
    .show(false)

// Exiting paste mode, now interpreting.

+-----------------------------------+
|CAST(date_trunc(DAY, ts) AS STRING)|
+-----------------------------------+
|0010-01-01 23:30:17                |
+-----------------------------------+
{code}
the result must be *0010-01-01 00:00:00*
Bug: Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/20 09:24
Affects Version/s: 2.4.5

This is a follow-up to the SPARK-30312. In SPARK-30312, the patch proposed to preserve path permission when truncating table. When setting up original ACLs, we need to set user/group/other permission as ACL entries too, otherwise if the path doesn't have default user/group/other ACL entries, ACL API will complain an error Invalid ACL: the user, group and other entries are required.
Bug: Nested pruning should not prune on required child outputs in Generate
Status: Closed | Priority: Major | Resolution: Cannot Reproduce
Created: 09/Feb/20 04:08
Affects Version/s: 3.0.0

We prune nested fields from Generate. If a child output is required in a top operator of Generate, we should not prune nested fields on it. Otherwise, the accessors on top operator could be unresolved.
Bug: Wrong result of to_utc_timestamp() on daylight saving day
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/20 17:03
Affects Version/s: 2.4.4

The to_utc_timestamp() function returns wrong result when:
* JVM system time zone is PST
* the session local time zone is UTC
* fromZone is Asia/Hong_Kong
for the local date '2019-11-03T12:00:00', the result must be '2019-11-03T04:00:00'
{code}
scala> import java.util.TimeZone
import java.util.TimeZone

scala> import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils._

scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> TimeZone.setDefault(getTimeZone("PST"))
scala> spark.conf.set("spark.sql.session.timeZone", "UTC")
scala> val df = Seq("2019-11-03T12:00:00").toDF("localTs")
df: org.apache.spark.sql.DataFrame = [localTs: string]

scala> df.select(to_utc_timestamp(col("localTs"), "Asia/Hong_Kong")).show
+-----------------------------------------+
|to_utc_timestamp(localTs, Asia/Hong_Kong)|
+-----------------------------------------+
|                      2019-11-03 03:00:00|
+-----------------------------------------+
{code}
 
See https://www.worldtimebuddy.com/?qm=1&lid=8,1819729,100&h=8&date=2019-11-2&sln=21-22
Bug: BroadcastExchangeExec does not fully honor "spark.broadcast.compress"
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 05/Feb/20 00:51
Affects Version/s: 2.4.0

Setting {{spark.broadcast.compress}} to false disables compression while sending broadcast variable to executors ([https://spark.apache.org/docs/latest/configuration.html#compression-and-serialization])

However this does not disable compression for any child relations sent by the executors to the driver. 

Setting spark.boradcast.compress to false should disable both sides of the traffic, allowing users to disable compression for the whole broadcast join for example.

[https://github.com/puneetguptanitj/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L89]

^here `executeCollectIterator` calls `getByteArrayRdd` which by default always gets a compressed stream