av9ash/GitBugs huggingface dataset:
Bug: Update `${sbtProject}/test:runMain` to `${sbtProject}/Test/runMain`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jul/22 11:05
Affects Version/s: 3.4.0

The following warnings are displayed when `build/sbt "core/test:runMain <this class>"` is executed.
{code:java}
[warn] sbt 0.13 shell syntax is deprecated; use slash syntax instead: core / Test / runMain {code}
Bug: Set missing column with defaultValue as constant in `ParquetColumnVector` 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Jul/22 06:52
Affects Version/s: 3.4.0

For the missing column with default value of ParquetColumnVector, setting isConstant to true can will prevent the `reset()` method from restoring the internal state of `WritableColumnVector`.

`OrcColumnarBatchReader` has done similar things to missing column.
Bug: Remove redundant trait: BeforeAndAfterAll & BeforeAndAfterEach & Logging
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jul/22 02:41
Affects Version/s: 3.3.0

SparkFunSuite declare as follow:
{code:java}
abstract class SparkFunSuite
extends AnyFunSuite
with BeforeAndAfterAll
with BeforeAndAfterEach
with ThreadAudit
with Logging
{code}
some suite extends SparkFunSuite and meanwhile with BeforeAndAfterAll or BeforeAndAfterEach or Logging, it is redundant.
Bug: spark UI about file size read
Status: Open | Priority: Minor | Resolution: 
Created: 08/Jul/22 06:23
Affects Version/s: 3.2.1

I run this sql 

*select * from cm_dw.dwd_client_install where proc_date=20220707;*

and then see the SQL page , it shows  the file size read: 2.1MB,it is right,the file on my storage is 2.1MB,

but The Stages tab  shows input size:4.3MB, I dont kown why it is not equal 2.1MB.

and I run this sql on spark 2.3.3,  Stages tabs shows input size:2.1MB,also right!

*!1.png!*

 

!2.png!

!3.png!
Bug: Upgrade joda-time from 2.10.13 to 2.10.14
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jul/22 11:57
Affects Version/s: 3.3.0

joda-time 2.10.14 was released, which supports the latest TZ database of 2022agtz.

release notes: https://www.joda.org/joda-time/changes-report.html#a2.10.14
Bug: Remove duplicate `.setAccessible(true)`  in `kvstore.KVTypeInfo`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jul/22 11:46
Affects Version/s: 3.4.0

{code:java}
    for (Method m : type.getDeclaredMethods()) {
      KVIndex idx = m.getAnnotation(KVIndex.class);
      if (idx != null) {
        checkIndex(idx, indices);
        Preconditions.checkArgument(m.getParameterTypes().length == 0,
          "Annotated method %s::%s should not have any parameters.", type.getName(), m.getName());
        m.setAccessible(true);
        indices.put(idx.value(), idx);
        m.setAccessible(true);
        accessors.put(idx.value(), new MethodAccessor(m));
      } {code}
The above code has duplicate calls to `.setAccessible(true)`.
Bug: Improve stats estimation for v2 tables
Status: In Progress | Priority: Minor | Resolution: 
Created: 05/Jul/22 08:21
Affects Version/s: 3.3.0

In case of v2 tables, connectors can bubble up both [sizeInBytes and rowCount |https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Statistics.java].

Presently, SizeInBytesOnlyStatsPlanVisitor, ommits propagating / estimating rowCount stats, some places like :
 * [CodePointer1|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/SizeInBytesOnlyStatsPlanVisitor.scala#L54-L58]
 * [CodePointer2 |https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/SizeInBytesOnlyStatsPlanVisitor.scala#L46-L47]

For the [non-cbo|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/LogicalPlanStats.scala#L34-L39] flow, as per my understanding, this can improve the stats estimation, since rowcount is indirectly used in places to estimate the size as well.
Bug: Remove `ColumnVectorUtils#populate(WritableColumnVector, InternalRow, int) `  method
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jul/22 07:41
Affects Version/s: 3.4.0

After SPARK-39638 and SPARK-39231
{code:java}
public static void populate(WritableColumnVector col, InternalRow row, int fieldIdx) {code}
in ColumnVectorUtils only used by {{ConstantColumnVectorBenchmark}} and {{ColumnVectorSuite}}
Bug: pyspark set "spark.hadoop.validateOutputSpecs" to false can't take effect
Status: Closed | Priority: Minor | Resolution: Not A Problem
Created: 04/Jul/22 08:04
Affects Version/s: 3.1.2

in pyspark,  set "spark.hadoop.validateOutputSpecs" to false can't take effect

example: 

spark = SparkSession.builder.config("spark.hadoop.validateOutputSpecs", "false").appName("PythonPi").getOrCreate()

rdd.saveAsTextFile(file_exist_path)

 

pyspark run  will report error:
|pyspark.sql.utils.AnalysisException: path hdfs://zjyprc-hadoop/user/s_ai_service/xxx already exists.|
Bug: YARN AM client should call the non-static setTokensConf method
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jul/22 16:53
Affects Version/s: 3.3.0

As part of SPARK-37205, Spark uses reflection to call {{setTokensConf}} method from YARN AM. However, it should invoke it as a non-static method instead of a static one.
Bug: Make Dataset.showString() public
Status: Open | Priority: Trivial | Resolution: 
Created: 08/Jul/22 11:04
Affects Version/s: 2.4.8

Currently, we have {{.show}} APIs on a Dataset, but they print directly to stdout.

But there are a lot of cases where we might need to get a String representation of the show output. For example
 * We have a logging framework to which we need to push the representation of a df
 * We have to send the string over a REST call from the driver
 * We want to send the string to stderr instead of stdout

For such cases, currently one needs to do a hack by changing the Console.out temporarily and catching the representation in a ByteArrayOutputStream or similar, then extracting the string from it.

Strictly only printing to stdout seems like a limiting choice. 

 

Solution:

We expose APIs to return the String representation back. We already have the .{{{}showString{}}} method internally.

 

We could mirror the current {{.show}} APIS with a corresponding {{.showString}} (and rename the internal private function to something else if required)
Bug: Remove Hive bucketing incompatibility doc
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 14/Jul/22 07:30
Affects Version/s: 3.4.0

We support Hive bucketing (with Hive hash function) started from Spark 3.3.0, we should also update the documentation to reflect the fact, that we are no longer incompatible with Hive bucketing.
Bug: Add a test case to read ORC table that requires type promotion
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Jul/22 12:18
Affects Version/s: 3.3.0

We can add a UT to test the scenario after the ORC-1205 release.

 

bin/spark-shell
{code:java}
spark.sql("set orc.stripe.size=10240")
spark.sql("set orc.rows.between.memory.checks=1")
spark.sql("set spark.sql.orc.columnarWriterBatchSize=1")
val df = spark.range(1, 1+512, 1, 1).map { i =>
    if( i == 1 ){
        (i, Array.fill[Byte](5 * 1024 * 1024)('X'))
    } else {
        (i,Array.fill[Byte](1)('X'))
    }
    }.toDF("c1","c2")
df.write.format("orc").save("file:///tmp/test_table_orc_t1")
spark.sql("create external table test_table_orc_t1 (c1 string ,c2 binary) location 'file:///tmp/test_table_orc_t1' stored as orc ")
spark.sql("select * from test_table_orc_t1").show() {code}
Querying this table will get the following exception
{code:java}
java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.orc.impl.TreeReaderFactory$TreeReader.nextVector(TreeReaderFactory.java:387)
        at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:740)
        at org.apache.orc.impl.ConvertTreeReaderFactory$StringGroupFromAnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:1069)
        at org.apache.orc.impl.reader.tree.StructBatchReader.readBatchColumn(StructBatchReader.java:65)
        at org.apache.orc.impl.reader.tree.StructBatchReader.nextBatchForLevel(StructBatchReader.java:100)
        at org.apache.orc.impl.reader.tree.StructBatchReader.nextBatch(StructBatchReader.java:77)
        at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1371)
        at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:84)
        at org.apache.orc.mapreduce.OrcMapreduceRecordReader.nextKeyValue(OrcMapreduceRecordReader.java:102)
        at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
 {code}
Bug: Avoid misleading vector example in ML doc
Status: Open | Priority: Trivial | Resolution: 
Created: 06/Jul/22 07:34
Affects Version/s: 2.0.0

Hi

In a recent discussion I discovered there was a misunderstanding in ML doc, taking its origin in chosen examples:  I think the clarity of documentation could be improved by only changing the values of vector in documentation.

Examples of misleading vector
{quote}{{dv = Vectors.dense(1.0, 0.0, 3.0);}}
{{{}sv = Vectors.sparse(3, new int[] {0, 2}, new double[] {1.0, 3.0});{}}}{{{}{}}}{{{}{}}}
{quote}
As you can see, the vector chosen in example has ordinal position of element as value, and has 3 elements wich is also value of 3rd element, which is also the size of the vector.  I understood this has been leading to confusion in the reader comprehesion (a newbie, not native english speaker) 

The example would gain in  understanding with example values  like this
{quote}{{{}dv = Vectors.dense(7.0, 0.0, 5.0);
{}}}{{{}sv = Vectors.sparse(3, new int[] {0, 2}, new double[] {7.0, 5.0});{}}}{{{}{}}}
{quote}
 

This ticket is a proposal to update docs using vector (1.0, 0.0, 3.0) into vector (7.0, 0.0, 5.0)
Bug: sparkstreaming对接kafka数据，在idea程序上出现警告信息并卡住不动的解决办法
Status: Resolved | Priority: Blocker | Resolution: Incomplete
Created: 17/Mar/21 03:30
Affects Version/s: 2.1.1

2021-03-17 11:16:56,711 WARN --- [ main] org.apache.spark.streaming.kafka010.KafkaUtils (line: 66) : overriding enable.auto.commit to false for executor
2021-03-17 11:16:56,714 WARN --- [ main] org.apache.spark.streaming.kafka010.KafkaUtils (line: 66) : overriding auto.offset.reset to none for executor
2021-03-17 11:16:56,714 WARN --- [ main] org.apache.spark.streaming.kafka010.KafkaUtils (line: 66) : overriding executor group.id to spark-executor-recommender
2021-03-17 11:16:56,715 WARN --- [ main] org.apache.spark.streaming.kafka010.KafkaUtils (line: 66) : overriding receive.buffer.bytes to 65536 see KAFKA-3135
Bug: Vectorized parquet reader needs synchronization among pages for column index
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 25/Mar/21 02:21
Affects Version/s: 3.2.0

the current implementation has a problem. the pages returned by `readNextFilteredRowGroup` may not be aligned, some columns may have more rows than others.

Parquet is using `org.apache.parquet.column.impl.SynchronizingColumnReader` with `rowIndexes` to make sure that rows are aligned. 

Currently `VectorizedParquetRecordReader` doesn't have such synchronizing among pages from different columns. Using `readNextFilteredRowGroup` may result in incorrect result.

 
I have attache an example parquet file. This file is generated with `spark.range(0, 2000).map(i => (i.toLong, i.toInt))` and the layout of this file is listed below.
row group 0
--------------------------------------------------------------------------------
_1:  INT64 SNAPPY DO:0 FPO:4 SZ:8161/16104/1.97 VC:2000 ENC:PLAIN,BIT_PACKED [more]...
_2:  INT32 SNAPPY DO:0 FPO:8165 SZ:8061/8052/1.00 VC:2000 ENC:PLAIN,BIT_PACKED [more]...

    _1 TV=2000 RL=0 DL=0
    ----------------------------------------------------------------------------
    page 0:  DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN ST:[no stats for  [more]... VC:500
    page 1:  DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN ST:[no stats for  [more]... VC:500
    page 2:  DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN ST:[no stats for  [more]... VC:500
    page 3:  DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN ST:[no stats for  [more]... VC:500

    _2 TV=2000 RL=0 DL=0
    ----------------------------------------------------------------------------
    page 0:  DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN ST:[no stats for  [more]... VC:1000
    page 1:  DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN ST:[no stats for  [more]... VC:1000
 
As you can see in the row group 0, column1 has 4 data pages each with 500 values and column 2 has 2 data pages with 1000 values each. 
If we want to filter the rows by values with _1 = 510 using columnindex, parquet will return the page 1 of column _1 and page 0 of column _2. Page 1 of column _1 starts with row 500, and page 0 of column _2 starts with row 0, and it will be incorrect if we simply read the two values as one row.
 
As an example, If you try filter with  _1 = 510 with column index on in current version, it will give you the wrong result
+---+---+
|_1 |_2 |
+---+---+
|510|10 |
+---+---+
And if turn columnindex off, you can get the correct result
+---+---+
|_1 |_2 |
+---+---+
|510|510|
+---+---+
Bug: Apply right-padding correctly for correlated subqueries
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Mar/21 06:57
Affects Version/s: 3.1.0

This ticket aim at  fixing the bug that does not apply right-padding for char types inside correlated subquries.
For example,  a query below returns nothing in master, but a correct result is `c`.
{code}
scala> sql(s"CREATE TABLE t1(v VARCHAR(3), c CHAR(5)) USING parquet")
scala> sql(s"CREATE TABLE t2(v VARCHAR(5), c CHAR(7)) USING parquet")
scala> sql("INSERT INTO t1 VALUES ('c', 'b')")
scala> sql("INSERT INTO t2 VALUES ('a', 'b')")
scala> val df = sql("""
  |SELECT v FROM t1
  |WHERE 'a' IN (SELECT v FROM t2 WHERE t2.c = t1.c )""".stripMargin)

scala> df.show()
+---+
|  v|
+---+
+---+

{code}

This is because `ApplyCharTypePadding`  does not handle the case above to apply right-padding into `'abc'`.
Bug: Full outer shuffled hash join when building left side produces wrong result
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 10/Mar/21 00:07
Affects Version/s: 3.1.0

For full outer shuffled hash join with building hash map on left side, and having non-equal condition, the join can produce wrong result.

The root cause is `boundCondition` in `HashJoin.scala` always assumes the left side row is `streamedPlan` and right side row is `buildPlan` (streamedPlan.output ++ buildPlan.output). This is valid assumption, except for full outer + build left case.

The fix is to correct `boundCondition` in `HashJoin.scala` to handle full outer + build left case properly. See reproduce in https://issues.apache.org/jira/browse/SPARK-32399?focusedCommentId=17298414&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17298414 .
Bug: Python type hints improvement
Status: Resolved | Priority: Critical | Resolution: Done
Created: 05/Mar/21 02:13
Affects Version/s: 3.1.2

We added PySpark type hints at SPARK-32681

However, looks like there are still many missing APIs to type. I maintain a project called [Koalas](https://github.com/databricks/koalas), and I found these errors https://gist.github.com/HyukjinKwon/9faabc5f2680b56007d71ef7cf0ad400
For example, {{pyspark.__version__}} and {{pyspark.sql.Column.contains}} are missing in the type hints.

I believe this is the same case to other projects that enables mypy in their project (presumably also given SPARK-34544).

This umbrella JIRA targets to identify such cases and improve Python type hints in PySpark.
Bug: Codegen compilation error for query with LIMIT operator and without AQE
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 19/Mar/21 00:46
Affects Version/s: 3.1.0

Example (reproduced in unit test):

 
{code:java}
  test("failed limit query") {
    withTable("left_table", "empty_right_table", "output_table") {
      spark.range(5).toDF("k").write.saveAsTable("left_table")
      spark.range(0).toDF("k").write.saveAsTable("empty_right_table")
         
      withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
        spark.sql("CREATE TABLE output_table (k INT) USING parquet")
        spark.sql(
          s"""
             |INSERT INTO TABLE output_table
             |SELECT t1.k FROM left_table t1
             |JOIN empty_right_table t2
             |ON t1.k = t2.k
             |LIMIT 3
             |""".stripMargin)
      }
    }
  }
{code}
Result:

 

https://gist.github.com/c21/ea760c75b546d903247582be656d9d66
Bug: Spark dependencies require scala 2.12.12
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 15/Mar/21 16:03
Affects Version/s: 3.1.1

In our application we're creating a spark session programmatically. The application is built using gradle.

After upgrading spark to 3.1.1 it no longer works, due to incompatible classes on driver and executor (namely: scala.lang.collections.immutable.WrappedArray.ofRef).



Turns out this was caused by different scala versions on driver vs. executor. While spark still comes with Scala 2.12.10, some of its dependencies in the gradle build require Scala 2.12.12:
{noformat}
Cannot find a version of 'org.scala-lang:scala-library' that satisfies the version constraints:
Dependency path '...' --> '...' --> 'org.scala-lang:scala-library:{strictly 2.12.10}'
Dependency path '...' --> 'org.apache.spark:spark-core_2.12:3.1.1' --> 'org.json4s:json4s-jackson_2.12:3.7.0-M5' --> 'org.scala-lang:scala-library:2.12.12' {noformat}
Bug: CVEs in the library dependencies
Status: Resolved | Priority: Critical | Resolution: Not A Problem
Created: 04/Mar/21 10:50
Affects Version/s: 3.1.1

Hi, I found various CVEs in dependency libraries bundled in pyspark shaded JARs, such as 
 * htrace-core4-4.1.0-incubating.jar:jackson-databind

with very old versions that are being flagged up in my vulnerability scans.

Are these already being addressed? If not, could you please update the references to newer versions?
|CVE ID|Type|Severity|Packages|Package Version|CVSS|Fix Status|
|CVE-2017-18640|java|high|org.yaml_snakeyaml|1.24|7.5|fixed in 1.26|
|CVE-2020-25649|java|high|com.fasterxml.jackson.core_jackson-databind|2.10.0|7.5|fixed in 2.10.5.1, 2.9.10.7, 2.6.7.4|
|CVE-2020-35491|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2020-35490|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2018-14718|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.7|
|CVE-2018-7489|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.5, 2.8.11.1, 2.7.9.3|
|CVE-2020-25649|java|high|com.fasterxml.jackson.core_jackson-databind|2.10.0|7.5|fixed in 2.10.5.1, 2.9.10.7, 2.6.7.4|
|CVE-2020-35491|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2020-35490|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2018-14718|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.7|
|CVE-2018-7489|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.5, 2.8.11.1, 2.7.9.3|
|CVE-2017-18640|java|high|org.yaml_snakeyaml|1.24|7.5|fixed in 1.26|
|CVE-2020-25649|java|high|com.fasterxml.jackson.core_jackson-databind|2.10.0|7.5|fixed in 2.10.5.1, 2.9.10.7, 2.6.7.4|
|CVE-2020-35491|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2020-35490|java|high|com.fasterxml.jackson.core_jackson-databind|2.4.0|8.1|fixed in 2.9.10.8|
|CVE-2018-14718|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.7|
|CVE-2018-7489|java|critical|com.fasterxml.jackson.core_jackson-databind|2.4.0|9.8|fixed in 2.9.5, 2.8.11.1, 2.7.9.3|
|CVE-2017-18640|java|high|org.yaml_snakeyaml|1.24|7.5|fixed in 1.26|
|CVE-2019-17195|java|critical|com.nimbusds_nimbus-jose-jwt|4.41.1|9.8|fixed in 7.9|

 

Similar issues, for reference:
 # CAMEL-14640
 # https://issues.apache.org/jira/browse/HADOOP-16690
 # ZEPPELIN-4657
 #
Bug: [K8S] Driver pod stuck in Running state after job completes
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/21 20:43
Affects Version/s: 3.0.2

I am running automated benchmarks in k8s, using spark-submit in cluster mode, so the driver runs in a pod.

When running with Spark 3.0.1 and 3.1.1 everything works as expected and I see the Spark context being shut down after the job completes.

However, when running with Spark 3.0.2 I do not see the context get shut down and the driver pod is stuck in the Running state indefinitely.

This is the output I see after job completion with 3.0.1 and 3.1.1 and this output does not appear with 3.0.2. With 3.0.2 there is no output at all after the job completes.
{code:java}
2021-03-05 20:09:24,576 INFO spark.SparkContext: Invoking stop() from shutdown hook
2021-03-05 20:09:24,592 INFO server.AbstractConnector: Stopped Spark@784499d0{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
2021-03-05 20:09:24,594 INFO ui.SparkUI: Stopped Spark web UI at http://benchmark-runner-3e8a38780400e0d1-driver-svc.default.svc:4040
2021-03-05 20:09:24,599 INFO k8s.KubernetesClusterSchedulerBackend: Shutting down all executors
2021-03-05 20:09:24,600 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
2021-03-05 20:09:24,609 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
2021-03-05 20:09:24,719 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
2021-03-05 20:09:24,736 INFO memory.MemoryStore: MemoryStore cleared
2021-03-05 20:09:24,738 INFO storage.BlockManager: BlockManager stopped
2021-03-05 20:09:24,744 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
2021-03-05 20:09:24,752 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
2021-03-05 20:09:24,768 INFO spark.SparkContext: Successfully stopped SparkContext
2021-03-05 20:09:24,768 INFO util.ShutdownHookManager: Shutdown hook called
2021-03-05 20:09:24,769 INFO util.ShutdownHookManager: Deleting directory /var/data/spark-67fa44df-e86c-463a-a149-25d95817ff8e/spark-a5476c14-c103-4108-b733-961400485d8a
2021-03-05 20:09:24,772 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-9d6261f5-4394-472b-9c9a-e22bde877814
2021-03-05 20:09:24,778 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2021-03-05 20:09:24,779 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
2021-03-05 20:09:24,779 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
 {code}
Bug: Spark SQL reads unnecessary nested fields (another type of pruning case)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/21 08:12
Affects Version/s: 3.1.1

Based on this [SPARK-29721|https://issues.apache.org/jira/browse/SPARK-29721] I found another nested fields pruning case.

Example:
1) Loading data

{code:scala}
val jsonStr = """{
 "items": [
   {"itemId": 1, "itemData": "a"},
   {"itemId": 2, "itemData": "b"}
 ]
}"""
val df = spark.read.json(Seq(jsonStr).toDS)
df.write.format("parquet").mode("overwrite").saveAsTable("persisted")
{code}

2) read query with explain

{code:scala}
val read = spark.table("persisted")
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", true)

read.select(explode($"items").as('item)).select($"item.itemId").explain(true)
// ReadSchema: struct<items:array<struct<itemData:string,itemId:bigint>>>
{code}