av9ash/GitBugs huggingface dataset:
Bug: The count result of the dimension table filed changes as `exector.memory` changes.
Status: Open | Priority: Major | Resolution: 
Created: 14/Sep/21 03:17
Affects Version/s: 2.1.3

hi~, every one!

Here‘s a very strange questions!!! 

The meaning of this sql is count the number of the specified columns in each table after joining the table。as follows:

 
{quote}SELECT cast(COUNT(DISTINCT tps.prod_siginst_id) AS STRING) AS siginst_cnt,
 cast(COUNT(DISTINCT qpl.list_id) AS STRING) AS list_cnt,
 cast(count(DISTINCT if(tb.brand_source=1,tps.prod_siginst_id,NULL)) AS STRING) AS domestic_siginst_cnt,
 cast(count(DISTINCT if(tb.brand_source=2,tps.prod_siginst_id,NULL)) AS STRING) AS import_siginst_cnt,
 cast(count(DISTINCT if(qpl.list_name NOT LIKE'%un_normal%',tps.prod_siginst_id,NULL)) AS STRING) AS standard_cnt,
 cast(count(DISTINCT if(qpl.list_name LIKE'%un_normal%',tps.prod_siginst_id,NULL)) AS STRING) AS nostandard_cnt
FROM tableA tbi
LEFT JOIN tableB tps ON tbi.prod_inst_id=tps.prod_inst_id
LEFT JOIN tableC qpl ON tbi.prod_type_id=qpl.list_id
LEFT JOIN tableD ON tps.brand_id=tb.brand_id
WHERE tbi.prod_status=1
 AND tbi.prod_sell_status=1
 AND tb.recommend_flag=1;
{quote}
 

and the phenomenon of the question is if i add memory for executor, the count result of the tableC field(list_id,list_name) will changes as well. until the executor‘s memory is big enough, the result doesn't change.

 

TableC is a dimensional table and the amount of data is fixed.

 

In my opinions, this job should failed rather than output an incorrect count result if executor is insufficient memory.

Could you please help me check whether this is a bug of spark itself or something wrong with my sql writing?

 

here is log of this job.
Bug: Structured streaming  maxOffsetsPerTrigger Invalidation
Status: Open | Priority: Major | Resolution: 
Created: 09/Sep/21 05:52
Affects Version/s: 2.4.5

Why does maxOffsetsPerTrigger not work when consuming Kafka, the task fails, and the yarn retries.

<dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
 <version>2.4.5</version>
 </dependency>

[https://stackoverflow.com/questions/55476504/restarting-spark-structured-streaming-job-consumes-millions-of-kafka-messages-an]
 There is a similar question here

!image-2021-09-09-13-57-21-175.png!
Bug: Cleanup the deprecated APIs and raise proper warning message.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/21 03:07
Affects Version/s: 3.3.0

Some of APIs in `missing/` marked as "Deprecated properties" or "Deprecated functions" are already removed in latest pandas.

Also we should check if the deprecated message is good enough or not, and refine the message to the proper way.
Bug: Replace collections.namedtuple() by typing.NamedTuple
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/21 01:47
Affects Version/s: 3.3.0

Per discussion under [https://github.com/apache/spark/pull/34133#discussion_r718833451,] we wanted to replace collections.namedtuple() by typing.NamedTuple.
Bug: spark-sql do NOT support  that select name expression as string type now
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 01/Sep/21 08:17
Affects Version/s: 3.1.0

The follow statement would throw an exception.


{code:java}
 sql("SELECT age as 'a', name as 'n' FROM VALUES (2, 'Alice'), (5, 'Bob') people(age, name)")
{code}



{code:java}
// Exception information
Error in query:
mismatched input ''a'' expecting {<EOF>, ';'}(line 1, pos 14)

== SQL ==
SELECT age as 'a', name as 'n' FROM VALUES (2, 'Alice'), (5, 'Bob') people(age, name)
--------------^^^
{code}
Bug: Ask users if they want to download and install SparkR in non Spark scripts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 06:38
Affects Version/s: 3.2.0

Spark is removed on CRAN because it disallows to write/download files into users' directories other then R session specific directories.

We should at least ask if users want to download and install it in this case.
Bug: pandas API on Spark: DataFrameGroupBy.apply raises an exception when it returns Series.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/21 23:43
Affects Version/s: 3.2.0

{{DataFrameGroupBy.apply}} without shortcut could raise an exception when it returns {{Series}}.

{code:python}
>>> ps.options.compute.shortcut_limit = 3
>>> psdf = ps.DataFrame(
...     {"a": [1, 2, 3, 4, 5, 6], "b": [1, 1, 2, 3, 5, 8], "c": [1, 4, 9, 16, 25, 36]},
...     columns=["a", "b", "c"],
... )
>>> psdf.groupby("b").apply(lambda x: x["a"])
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
...
ValueError: Length mismatch: Expected axis has 2 elements, new values have 3 elements
{code}
Bug: Sha2 with bit_length 512 not being tested
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/21 04:38
Affects Version/s: 3.2.0

Looking at [https://github.com/apache/spark/commit/6c6291b3f6ac13b8415b87b2b741a9cd95bc6c3b] for https://issues.apache.org/jira/browse/SPARK-36836, it's clear that 512 bits are supported

{{bitLength match {}}
 {{[...]}}
 {{  case 512 =>}}
 {{    UTF8String.fromString(DigestUtils.sha512Hex(input))}}

resp.

{{nullSafeCodeGen(ctx, ev, (eval1, eval2) => {}}
 {{    [...]}}
 {{    else if ($eval2 == 512) {}}
 {{       ${ev.value} =}}
 {{         UTF8String.fromString($digestUtils.sha512Hex($eval1));}}

but the test claims it is unsupported:

{{// unsupported bit length}}
 {{checkEvaluation(Sha2(Literal.create(null, BinaryType), Literal(1024)), null)}}
 {{checkEvaluation(Sha2(Literal.create(null, BinaryType), Literal(512)), null)}}

To avoid a similar fate as SPARK-36836, tests should be added.
  
 CC [~richardc-db]
Bug: Respect `spark.sql.parquet.filterPushdown` by explain() for DSv2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/21 06:57
Affects Version/s: 3.3.0

When filters pushdown for parquet is disabled via the SQL config spark.sql.parquet.filterPushdown, explain() still outputs pushed down filters:

{code}
== Parsed Logical Plan ==
'Filter ('c0 = 1)
+- RelationV2[c0#7] parquet file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-ff7e9a24-fd4e-4981-9c75-e1bcde78e91a

== Analyzed Logical Plan ==
c0: int
Filter (c0#7 = 1)
+- RelationV2[c0#7] parquet file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-ff7e9a24-fd4e-4981-9c75-e1bcde78e91a

== Optimized Logical Plan ==
Filter (isnotnull(c0#7) AND (c0#7 = 1))
+- RelationV2[c0#7] parquet file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-ff7e9a24-fd4e-4981-9c75-e1bcde78e91a

== Physical Plan ==
*(1) Filter (isnotnull(c0#7) AND (c0#7 = 1))
+- *(1) ColumnarToRow
   +- BatchScan[c0#7] ParquetScan DataFilters: [isnotnull(c0#7), (c0#7 = 1)], Format: parquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-ff..., PartitionFilters: [], PushedFilters: [IsNotNull(c0), EqualTo(c0,1)], ReadSchema: struct<c0:int>, PushedFilters: [IsNotNull(c0), EqualTo(c0,1)] RuntimeFilters: []
{code}
See PushedFilters: [IsNotNull(c0), EqualTo(c0,1)]
Bug: Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 28/Sep/21 16:41
Affects Version/s: 3.1.2

In one of our jobs we perform the following operation:
{code:scala}
val df = /* some expensive multi-table/multi-stage join */
val numPartitions = df.rdd.getNumPartitions
df.repartition(x).write.....
{code}

With AQE enabled, we found that the expensive stages were being run twice causing significant performance regression after enabling AQE; once when calling {{df.rdd}} and again when calling {{df.write}}.

A more concrete example:
{code:scala}
scala> sql("SET spark.sql.adaptive.enabled=true")
res0: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> sql("SET spark.sql.autoBroadcastJoinThreshold=-1")
res1: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> val df1 = spark.range(10).withColumn("id2", $"id")
df1: org.apache.spark.sql.DataFrame = [id: bigint, id2: bigint]

scala> val df2 = df1.join(spark.range(10), "id").join(spark.range(10), "id").join(spark.range(10), "id")
df2: org.apache.spark.sql.DataFrame = [id: bigint, id2: bigint]

scala> val df3 = df2.groupBy("id2").count()
df3: org.apache.spark.sql.DataFrame = [id2: bigint, count: bigint]

scala> df3.rdd.getNumPartitions
res2: Int = 10                                                    (0 + 16) / 16]

scala> df3.repartition(5).write.mode("overwrite").orc("/tmp/orc1")
{code}

In the screenshot below, you can see that the first 3 stages (0 to 4) were rerun again (5 to 9).

I have two questions:
1) Should calling df.rdd trigger actual job execution when AQE is enabled?
2) Should calling df.write later cause rerun of the stages? If df.rdd has already partially executed the stages, shouldn't it reuse the result from previous stages?
Bug: Create the external hive table for HBase failed 
Status: Open | Priority: Major | Resolution: 
Created: 27/Sep/21 03:54
Affects Version/s: 3.1.2

We use follow sql to create hive external table , which read from hbase
{code:java}
CREATE EXTERNAL TABLE if not exists dev.sanyu_spotlight_headline_material(
   rowkey string COMMENT 'HBase主键',
   content string COMMENT '图文正文')
USING HIVE   
ROW FORMAT SERDE
   'org.apache.hadoop.hive.hbase.HBaseSerDe'
 STORED BY
   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES (
   'hbase.columns.mapping'=':key, cf1:content'
)
 TBLPROPERTIES (
   'hbase.table.name'='spotlight_headline_material'
 );
{code}
But the sql failed in Spark 3.1.2, which throw this exception
{code:java}
21/09/27 11:44:24 INFO scheduler.DAGScheduler: Asked to cancel job group 26d7459f-7b58-4c18-9939-5f2737525ff2
21/09/27 11:44:24 ERROR thriftserver.SparkExecuteStatementOperation: Error executing query with 26d7459f-7b58-4c18-9939-5f2737525ff2, currentState RUNNING,
org.apache.spark.sql.catalyst.parser.ParseException:
Operation not allowed: Unexpected combination of ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' and STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITHSERDEPROPERTIES('hbase.columns.mapping'=':key, cf1:content')(line 5, pos 0)
{code}
this check was introduced from this change: [https://github.com/apache/spark/pull/28026]

 

Could anyone gave the introduction how to create the external table for hbase in spark3 now ?
Bug: Inline most of type hint files under pyspark/sql/pandas folder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/21 20:43
Affects Version/s: 3.3.0

Inline type hint files under {{pyspark/sql/pandas}} folder, except for {{pyspark/sql/pandas/functions.pyi}} and files under {{pyspark/sql/pandas/_typing}}.
 * Since the file contains a lot of overloads, we should revisit and manage it separately.
 * We can't inline files under {{pyspark/sql/pandas/_typing}} because it includes new syntax for type hints.
Bug: Parquet reader fails on load of ANSI interval when off-heap is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/21 19:09
Affects Version/s: 3.3.0

When off-heap column vector is enabled, parquet reader fails. The example below portraits the issue:

{code:scala}
scala> Seq(java.time.Period.of)
scala> val df = Seq(java.time.Period.ofMonths(1)).toDF
df: org.apache.spark.sql.DataFrame = [value: interval year to month]

scala> df.write.parquet("/Users/maximgekk/tmp/parquet_offheap")
scala> spark.conf.set("spark.sql.columnVector.offheap.enabled", true)

scala> spark.read.parquet("/Users/maximgekk/tmp/parquet_offheap")
res2: org.apache.spark.sql.DataFrame = [value: interval year to month]

scala> spark.read.parquet("/Users/maximgekk/tmp/parquet_offheap").show()
21/09/25 22:09:03 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.lang.RuntimeException: Unhandled YearMonthIntervalType(0,1)
	at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.reserveInternal(OffHeapColumnVector.java:562)
	at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.<init>(OffHeapColumnVector.java:75)
	at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.allocateColumns(OffHeapColumnVector.java:53)
	at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.allocateColumns(OffHeapColumnVector.java:42)
{code}
Bug: Incorrect writing the string, containing symbols like "\" to Hive 
Status: Open | Priority: Major | Resolution: 
Created: 19/Sep/21 20:53
Affects Version/s: 2.3.0

After writing the strings, containing symbol like "\" to Hive, the result record in HiveTable doesn't contain that symbol. It happens when using the standart method of pyspark.sql.readwriter.DataFrameWriter saveAsTable as well as insertInto.

For example, running the query

spark.sql("select '\d\{4}' as code").write.saveAsTable('db.table')

I've got the next result in Hive:

spark.table('db.table').collect()[0][0]

>>"d\{4}"

But expected the next 

>> "\d\{4}"

Spark version : '2.3.0.2.6.5.0-292'
Bug: Does Apache Spark 3 support GPU usage for Spark RDDs?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 21/Sep/21 17:57
Affects Version/s: 3.1.2

I am currently trying to run genomic analyses pipelines using [Hail|https://hail.is/](library for genomics analyses written in python and Scala). Recently, Apache Spark 3 was released and it supported GPU usage.

I tried [spark-rapids|https://nvidia.github.io/spark-rapids/] library start an on-premise slurm cluster with gpu nodes. I was able to initialise the cluster. However, when I tried running hail tasks, the executors keep getting killed.

On querying in Hail forum, I got the response that
{quote}That’s a GPU code generator for Spark-SQL, and Hail doesn’t use any Spark-SQL interfaces, only the RDD interfaces.
{quote}
So, does Spark3 not support GPU usage for RDD interfaces?
Bug: Explain Formatted has Duplicated Node IDs with InMemoryRelation Present
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/21 21:37
Affects Version/s: 3.1.2

When a query contains an InMemoryRelation, the output of Explain Formatted will contain duplicate node IDs.


{code:java}
== Physical Plan ==
AdaptiveSparkPlan (14)
+- == Final Plan ==
   * BroadcastHashJoin Inner BuildLeft (9)
   :- BroadcastQueryStage (5)
   :  +- BroadcastExchange (4)
   :     +- * Filter (3)
   :        +- * ColumnarToRow (2)
   :           +- InMemoryTableScan (1)
   :                 +- InMemoryRelation (2)
   :                       +- * ColumnarToRow (4)
   :                          +- Scan parquet default.t1 (3)
   +- * Filter (8)
      +- * ColumnarToRow (7)
         +- Scan parquet default.t2 (6)
+- == Initial Plan ==
   BroadcastHashJoin Inner BuildLeft (13)
   :- BroadcastExchange (11)
   :  +- Filter (10)
   :     +- InMemoryTableScan (1)
   :           +- InMemoryRelation (2)
   :                 +- * ColumnarToRow (4)
   :                    +- Scan parquet default.t1 (3)
   +- Filter (12)
      +- Scan parquet default.t2 (6)
{code}
Bug: The log could not get the correct line number
Status: Open | Priority: Major | Resolution: 
Created: 16/Sep/21 10:19
Affects Version/s: 2.4.6

INFO 18:16:46 [Thread-1] org.apache.spark.internal.Logging$class.logInfo({color:#FF0000}Logging.scala:54{color}) MemoryStore cleared
 INFO 18:16:46 [Thread-1] org.apache.spark.internal.Logging$class.logInfo({color:#FF0000}Logging.scala:54{color}) BlockManager stopped
 INFO 18:16:46 [Thread-1] org.apache.spark.internal.Logging$class.logInfo({color:#FF0000}Logging.scala:54{color}) BlockManagerMaster stopped
 INFO 18:16:46 [dispatcher-event-loop-0] org.apache.spark.internal.Logging$class.logInfo({color:#FF0000}Logging.scala:54{color}) OutputCommitCoordinator stopped!
 INFO 18:16:46 [Thread-1] org.apache.spark.internal.Logging$class.logInfo({color:#FF0000}Logging.scala:54{color}) Successfully stopped SparkContext
 INFO 18:16:46 [Thread-1] org.apache.spark.internal.Logging$class.logInfo({color:#FF0000}Logging.scala:54{color}) Shutdown hook called

all are : {color:#FF0000}Logging.scala:54{color}
Bug: Fix the uts to check the parquet compression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/21 03:23
Affects Version/s: 3.3.0

In my own test env. I import a wrong jar version about zstd-jni-1.4.4-3.jar with the parquet version 1.12.0, when I pass the full uts and release a spark version to my env , using the parquet table with compression zstd, it throw a "NoClassDefFoundError" Exception. And I check the parquet source code, find the right zstd version for the 1.12.0 is 1.4.9-1. Once I upgrade the version about zstd-jni, it work well. 

!https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2021/8/14/xwx915262/ed1735e06a684bf19cf1d1a68a130b6a/image.png!

But I think we need check for all plugin compression algorithm can work well when we upgrade the version about those compression jars and parquet version.
Bug: Error when list of data type tuples has len = 1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/21 09:37
Affects Version/s: 3.3.0


{code:python}
>>> ps.DataFrame[("a", int), [int]]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, int]

>>> ps.DataFrame[("a", int), [("b", int)]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dgd/spark/python/pyspark/pandas/frame.py", line 11998, in __class_getitem__
    return create_tuple_for_frame_type(params)
  File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line 685, in create_tuple_for_frame_type
    return Tuple[extract_types(params)]
  File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line 755, in extract_types
    return (index_type,) + extract_types(data_types)
  File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line 770, in extract_types
    raise TypeError(
TypeError: Type hints should be specified as one of:
  - DataFrame[type, type, ...]
  - DataFrame[name: type, name: type, ...]
  - DataFrame[dtypes instance]
  - DataFrame[zip(names, types)]
  - DataFrame[index_type, [type, ...]]
  - DataFrame[(index_name, index_type), [(name, type), ...]]
  - DataFrame[dtype instance, dtypes instance]
  - DataFrame[(index_name, index_type), zip(names, types)]
However, got [('b', <class 'int'>)].

{code}
Bug: Spark SQL DDL does not recognize fs.s3.impl implied filesystem in LOCATION tag
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 15/Sep/21 14:55
Affects Version/s: 3.0.2

CREATE TABLE with the LOCATION tag via SPARK SQL DDL does not seem to recognize the fileSystem implied by setting fs.s3.impl.

The environment setting is set to:
|spark.hadoop.fs.s3.impl|org.apache.hadoop.fs.s3a.S3AFileSystem|

The following spark.sql ddl raises an exception:
 spark.sql("""CREATE TABLE local.db.table (
 col1 string,
 col2 string,
 )
 LOCATION 's3://bucket/prefix'
 """)
 Exception: 

Py4JJavaError: An error occurred while calling o86.sql. : org.apache.iceberg.exceptions.CommitStateUnknownException: Got exception: org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme "s3"

This is inconsistent with other spark functions such as spark.read.parquet("s3://bucket/prefix") which succeeds with the same configuration
Bug: Partition filter of DataSourceV2ScanRelation can not push down when select none dataSchema from FileScan
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/Sep/21 06:36
Affects Version/s: 3.1.2

In PruneFileSourcePartitions rule, the FileScan::withFilters is called to push down partition prune filter(and this is the only place this function can be called), but it has a constraint that “scan.readDataSchema.nonEmpty” 
 [source code here|https://github.com/apache/spark/blob/de351e30a90dd988b133b3d00fa6218bfcaba8b8/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala#L114]
 We use spark sql in custom catalog and execute the count sql like: select count( * ) from catalog.db.tbl where dt=‘0812’ (also in other sqls if we not select any col reference to tbl), in which dt is a partition key.

In this case the scan.readDataSchema is empty indeed and no scan partition prune performed, which cause scan all partition at last.
Bug: Support to specify index type and name in pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Done
Created: 10/Sep/21 06:34
Affects Version/s: 3.3.0

See https://koalas.readthedocs.io/en/latest/user_guide/typehints.html.

pandas API on Spark currently there's no way to specify the index type and name in the output when you apply an arbitrary function, which forces to create the default index:

{code}
>>> def transform(pdf) -> pd.DataFrame["id": int, "A": int]:
...     pdf['A'] = pdf.id + 1
...     return pdf
...
>>> ps.range(5).koalas.apply_batch(transform)
{code}

{code}
   id   A
0   0   1
1   1   2
2   2   3
3   3   4
4   4   5
{code}

We should have a way to specify the index.
Bug: ERROR exchange.BroadcastExchangeExec: Could not execute broadcast in 300 secs
Status: Open | Priority: Major | Resolution: 
Created: 30/Sep/21 07:23
Affects Version/s: 2.4.0

While running Spark application, if there are no further resources to launch executors, Spark application is failed after 5 mins with below exception.
{code:java}
21/09/24 06:12:45 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
...
21/09/24 06:17:29 ERROR exchange.BroadcastExchangeExec: Could not execute broadcast in 300 secs.
java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]
...
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
	at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:146)
	... 71 more
21/09/24 06:17:30 INFO spark.SparkContext: Invoking stop() from shutdown hook
{code}
*Expectation* should be either needs to be throw proper exception saying *"there are no further resources to run the application"* or it needs to be *"wait till it get resources"*.

To reproduce the issue we have used following sample code.

*PySpark Code (test_broadcast_timeout.py):*
{code:java}
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Test Broadcast Timeout").getOrCreate()
t1 = spark.range(5)
t2 = spark.range(5)

q = t1.join(t2,t1.id == t2.id)
q.explain
q.show(){code}
*Spark Submit Command:*
{code:java}
spark-submit --executor-memory 512M test_broadcast_timeout.py{code}
 Note: We have tested same code in Spark 3.1, we are able to reproduce the issue in Spark3 as well.
Bug: Support Parquet v2 data page encodings for the vectorized path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/21 18:22
Affects Version/s: 3.3.0

Currently Spark only support Parquet V1 encodings (i.e., PLAIN/DICTIONARY/RLE) in the vectorized path, and throws exception otherwise:
{code}
java.lang.UnsupportedOperationException: Unsupported encoding: DELTA_BYTE_ARRAY
{code}

It will be good to support v2 encodings too, including DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY as well as BYTE_STREAM_SPLIT as listed in https://github.com/apache/parquet-format/blob/master/Encodings.md