av9ash/GitBugs huggingface dataset:
Bug: Unify the pushed operator checking between FileSource test suite and JDBC test suite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Apr/22 22:16
Affects Version/s: 3.3.0

In JDBCV2Suite, we use checkPushedInfo to check the pushed down operators. Will do the same for FileSourceAggregatePushDownSuite


{code:java}
  private def checkPushedInfo(df: DataFrame, expectedPlanFragment: String): Unit = {
    df.queryExecution.optimizedPlan.collect {
      case _: DataSourceV2ScanRelation =>
        checkKeywordsExistsInExplain(df, expectedPlanFragment)
    }
  }
{code}
Bug: Update the Oracle docker image version used for test and integration
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Apr/22 08:31
Affects Version/s: 3.2.1

This proposes to update the Docker image used for integration tests and builds from Oracle XE version 18.4.0 to Oracle XE version 21.3.0.

Currently Oracle XE version 18.4.0 is being used. Oracle 18c support has ended in 2021. Oracle 21c is the latest release of the Oracle RDBMS.
Bug: Clean up useless security configs in `TransportConf`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Apr/22 06:45
Affects Version/s: 3.4.0

There are some configuration items in `TransportConf` that are no longer used after [Update Spark key negotiation protocol|https://github.com/apache/spark/commit/3b0dd14f1c5dd033ad0a6295baa288eda9dfe10a]
 * spark.network.crypto.keyFactoryAlgorithm
 * spark.network.crypto.keyLength
 * spark.network.crypto.ivLength
 * spark.network.crypto.keyAlgorithm
Bug: Use `Map.values()` instead of `Map.entrySet()` in scenarios that do not use `keys`
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/Apr/22 08:42
Affects Version/s: 3.4.0

Some code in Spark use `Map.entrySet()` but not use `keys`, can use   `Map.values()` instead of
Bug: Improve error log readability in OrcUtils.requestedColumnIds
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Apr/22 07:21
Affects Version/s: 3.2.1

In OrcUtils#requestedColumnIds sometimes it fails because orcFieldNames.length > dataSchema.length, the log is not very clear.
{code:java}
java.lang.AssertionError: assertion failed: The given data schema struct<field1:int> has less fields than the actual ORC physical schema, no idea which columns were dropped, fail to read. {code}
Bug: Aggregate aliases override field names in ResolveAggregateFunctions
Status: In Progress | Priority: Trivial | Resolution: 
Created: 19/Apr/22 17:26
Affects Version/s: 3.2.1

Spark versions before 3.1.x can query the following SQL:

{code:java}
select sum(id) as id  from range(10) group by id order by sum(id);{code}
{code:java}
Error in query: Resolved attribute(s) id#0L missing from id#1L in operator !Aggregate [id#1L], [sum(id#1L) AS id#0L, sum(id#0L) AS sum(id#0L)#4L]. Attribute(s) with the same name appear in the operation: id. Please check if the right attribute(s) are used.;
Project [id#0L]
+- Sort [sum(id#0L)#4L ASC NULLS FIRST], true
   +- !Aggregate [id#1L], [sum(id#1L) AS id#0L, sum(id#0L) AS sum(id#0L)#4L]
      +- Range (0, 10, step=1, splits=None) {code}
Bug: dropFieldIfAllNull option does not work for empty JSON struct
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Apr/22 02:41
Affects Version/s: 3.2.1

As stated in the doc, 
{quote}dropFieldIfAllNull

Whether to ignore column of all null values or empty array/struct during schema inference.

 
{quote}
But when I try this, 

 
{code:java}
String json = "{\"field1\":\"value1\", \"field2\":{}}";
JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());
JavaRDD<String> jrdd = jsc.parallelize(Arrays.asList(json));
Dataset<Row> df = spark.read().option("dropFieldIfAllNull", "false").json(jrdd);
df.printSchema();
{code}
 

I get this, 
{code:java}
root
 |-- field1: string (nullable = true){code}
Notice field2 is still missing even when dropFieldIfAllNull is set to false, so apparently, this option does not work for empty struct.

This is due to SPARK-8093, the empty struct will be dropped anyway.

I think we should update the doc, otherwise it would be confusing.

I can make a patch for this.
Bug: Disable push based shuffle when IO encryption is enabled or serializer is not relocatable
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 09/Sep/21 17:23
Affects Version/s: 3.2.0

Push based shuffle is not compatible with io encryption or non-relocatable serialization.
This is similar to SPARK-34790

We have to disable push based shuffle if either of these two are true.
Bug: Partition columns are overly eagerly parsed as dates
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 27/Sep/21 07:42
Affects Version/s: 3.3.0

I have an input directory with subdirs:
* hour=2021-01-01T00
* hour=2021-01-01T01
* hour=2021-01-01T02
* ...

in spark 3.1 the 'hour' column is parsed as a string type, but in 3.2 RC it is parsed as date type and the hour part is lost.
Bug: Disable batch fetch for a shuffle when push based shuffle is enabled
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 29/Sep/21 20:22
Affects Version/s: 3.2.0

When push based shuffle is enabled, efficient fetch of merged mapper shuffle output happens.
Unfortunately, this currently interacts badly with spark.sql.adaptive.fetchShuffleBlocksInBatch, potentially causing shuffle fetch to hang and/or duplicate data to be fetched, causing correctness issues.

Given batch fetch does not benefit spark stages reading merged blocks when push based shuffle is enabled, ShuffleBlockFetcherIterator.doBatchFetch can be disabled when push based shuffle is enabled.


Thx to [~Ngone51] for surfacing this issue.
+CC [~Gengliang.Wang]
Bug: Spark 3.2.0 POMs are no longer "dependency reduced"
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Sep/21 23:19
Affects Version/s: 3.2.0

It looks like Spark 3.2.0's POMs are no longer "dependency reduced". As a result, applications may pull in additional unnecessary dependencies when depending on Spark.

Spark uses the Maven Shade plugin to create effective POMs and to bundle shaded versions of certain libraries with Spark (namely, Jetty, Guava, and JPPML). [By default|https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#createDependencyReducedPom], the Maven Shade plugin generates simplified POMs which remove dependencies on artifacts that have been shaded.

SPARK-33212 / [b6f46ca29742029efea2790af7fdefbc2fcf52de|https://github.com/apache/spark/commit/b6f46ca29742029efea2790af7fdefbc2fcf52de] changed the configuration of the Maven Shade plugin, setting {{createDependencyReducedPom}} to {{false}}.

As a result, the generated POMs now include compile-scope dependencies on the shaded libraries. For example, compare the {{org.eclipse.jetty}} dependencies in:
 * Spark 3.1.2: [https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/3.1.2/spark-core_2.12-3.1.2.pom]
 * Spark 3.2.0 RC2: [https://repository.apache.org/content/repositories/orgapachespark-1390/org/apache/spark/spark-core_2.12/3.2.0/spark-core_2.12-3.2.0.pom]

I think we should revert back to generating "dependency reduced" POMs to ensure that Spark declares a proper set of dependencies and to avoid "unknown unknown" consequences of changing our generated POM format.

/cc [~csun]
Bug: Fail to load Lz4 codec
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 04/Sep/21 21:45
Affects Version/s: 3.2.0

Currently we use Hadop 3.3.1's shaded client libraries. Lz4 is a provided dependency in Hadoop Common 3.3.1 for Lz4Codec. But it isn't excluded from relocation in these libraries. So to use lz4 as Parquet codec, we will hit the exception even we include lz4 as dependency.
{code:java}
[info]   Cause: java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/net/jpountz/lz4/LZ4Factory                                                                                            
[info]   at org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:66)
[info]   at org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)                                                                                                         
[info]   at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)                                                                                                          
[info]   at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)                                                                                                          
 {code}
 

I already submitted a PR to Hadoop to fix it. Before it is released, at Spark side, we either downgrade to 3.3.0 or revert back to non-shaded hadoop client library.
Bug: FinalizeShuffleMerge fails with an exception due to attempt id not matching
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 15/Sep/21 23:53
Affects Version/s: 3.2.0

As part of driver request to external shuffle services (ESS) to finalize the merge, it also passes its [application attempt id|https://github.com/apache/spark/blob/3f09093a21306b0fbcb132d4c9f285e56ac6b43c/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java#L180] so that ESS can validate the request is from the correct attempt.
This attempt id is fetched from the TransportConf passed in when creating the [ExternalBlockStoreClient|https://github.com/apache/spark/blob/67421d80b8935d91b86e8cd3becb211fa2abd54f/core/src/main/scala/org/apache/spark/SparkEnv.scala#L352] - and the transport conf leverages a [cloned copy|https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/core/src/main/scala/org/apache/spark/network/netty/SparkTransportConf.scala#L47] of the SparkConf passed to it.


Application attempt id is set as part of SparkContext [initialization|https://github.com/apache/spark/blob/67421d80b8935d91b86e8cd3becb211fa2abd54f/core/src/main/scala/org/apache/spark/SparkContext.scala#L586].
But this happens after driver SparkEnv has [already been created|https://github.com/apache/spark/blob/67421d80b8935d91b86e8cd3becb211fa2abd54f/core/src/main/scala/org/apache/spark/SparkContext.scala#L460].

Hence the attempt id that ExternalBlockStoreClient uses will always end up being -1 : which will not match the attempt id at ESS (which is based on spark.app.attempt.id) : resulting in merge finalization to always fail (" java.lang.IllegalArgumentException: The attempt id -1 in this FinalizeShuffleMerge message does not match with the current attempt id 1 stored in shuffle service for application ...")
Bug: spark.read.parquet loads empty dataset
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 08/Sep/21 18:50
Affects Version/s: 3.2.0

Here's a parquet file Spark 3.2/master can't read properly.

The file was stored by pandas and must contain 3650 rows, but Spark 3.2/master returns an empty dataset.
{code:python}
>>> import pandas as pd
>>> len(pd.read_parquet('/path/to/example.parquet'))
3650

>>> spark.read.parquet('/path/to/example.parquet').count()
0
{code}
I guess it's caused by the parquet 1.12.0.

When I reverted two commits related to the parquet 1.12.0 from branch-3.2:
 - [https://github.com/apache/spark/commit/e40fce919ab77f5faeb0bbd34dc86c56c04adbaa]
 - [https://github.com/apache/spark/commit/cbffc12f90e45d33e651e38cf886d7ab4bcf96da]

it reads the data successfully.

We need to add some workaround, or revert the commits.
Bug: BlockManager re-registration is broken due to deferred removal of BlockManager 
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 09/Sep/21 02:28
Affects Version/s: 3.0.3

Due to the deferred removal of BlockManager (introduced in SPARK-35011), an expected BlockManager re-registration could be refused as the inactive BlockManager still exists in the map `blockManagerInfo`:

https://github.com/apache/spark/blob/9cefde8db373a3433b7e3ce328e4a2ce83b1aca2/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L551
Bug: [SQL] Regression in AQEShuffleReadExec
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 03/Sep/21 23:51
Affects Version/s: 3.2.0

I am currently testing the RAPIDS Accelerator for Apache Spark with the Spark 3.2 release candidate and there is a regression in AQEShuffleReadExec where it now throws an exception if the shuffle's output partitioning does not match a specific list of schemes.

The problem can be solved by returning UnknownPartitioning, as it already does in some cases, rather than throwing an exception.
Bug: Can't use SSL with spark on kubernetes on service level
Status: Open | Priority: Critical | Resolution: 
Created: 23/Sep/21 12:17
Affects Version/s: 3.0.0

Currently it seems impossible to create the correct cert for driver's pod because of the random naming of the service.

I would like to use ssl on spark Ui which will be accessed by other pods using the driver's service.
{code:java}
"spark.ssl.enabled"=true
"spark.ssl.keyStore"=my-spark.jks
"spark.ssl.keyStorePassword"=mypassword
..etc..{code}
At this point we already have to know the domain for the cert.

Which we don't because it will be generated at time when the driver pod generated.
{code:java}
my-application-75f3654hj76gb67n-driver
my-application-75f3654hj76gb67n-driver-svc{code}
So SSL handshake will fail with :
{code:java}
" SSL: no alternative certificate subject name matches target host name my-application-75f3654hj76gb67n-driver-svc{code}
I tried to mod the pod name with:
{code:java}
 spark.kubernetes.driver.pod.name{code}
but it only affects the pod name and not the service name

If it is neither a bug nor a missed feature then please guide me how to use SSL when hitting the driver's service (or how to define fixed name service like for pods).

----------------------------------------------------------------------------------------------------------------------------------------------------------------

I found a *partial solution* using wildcards for domain inside the cert, but because it only works on subdomain level I have to refer the service with :
 <POD_NAME>-*-driver-svc.<NS>.svc as alternatedomain inside the cert
 and using it with the namespace , svc added just to conform the wildcard's rule subdomain restriction
Bug: PythonRunner failed should pass error message to ApplicationMaster too
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Sep/21 06:10
Affects Version/s: 3.2.0

In current pyspark, stderr and stdout are print together, if py-script failed, will only throw a SparkUserAppsException with exit code 1. Then pass this error to AM.
In cluster mode, client side only show 
{code}
User application exited with 1.
{code}
Without correct error message.  Then user need to find why their job failed.
Bug: Supports Dynamic Table Options for Spark SQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/21 03:56
Affects Version/s: 3.1.2

Now a DataFrame API user can implement dynamic options through the _DataFrameReader$option_ method, but Spark SQL users cannot use.
{code:java}
DataFrameReader/AstBuilder -> UnresolvedRelation$options -> DataSourceV2Relation$options -> SupportsRead$newScanBuilder(options)
{code}
 
 The table options were persisted to the Catalog and if we want to modify that, we should use another DDL like "_ALTER TABLE ..._". But there are some cases that user want to modify the table options dynamically just in the query:
 * JDBCTable set _fetchsize_ according to the actual situation of the table
 * IcebergTable support time travel
{code:java}
spark.read
    .option("snapshot-id", 10963874102873L)
    .format("iceberg")
    .load("path/to/table"){code}

These parameters setting is very common and ad-hoc, setting them flexibly would promote the user experience with Spark SQL especially for Now we support catalog expansion.
Bug: Handle HDFS read inconsistencies on Spark when observer Namenode is used
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/21 17:54
Affects Version/s: 3.2.0

In short, with HDFS HA and with the use of [Observer Namenode|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html] the read-after-write consistency is only available when both the write and the read happens from the same client.

But if the write happens on executor and the read happens on the driver, then the reads would be inconsistent causing application failure issues. This can be fixed by calling `FileSystem.msync` before making any read calls where the client thinks the write could have possibly happened elsewhere.

This issue is discussed in greater detail in this [discussion|https://mail-archives.apache.org/mod_mbox/spark-dev/202108.mbox/browser]
Bug: Add LauncherProtocol to K8s resource manager client
Status: Open | Priority: Major | Resolution: 
Created: 23/Sep/21 12:16
Affects Version/s: 3.1.2

Refer to: org.apache.spark.deploy.yarn.Client
{code:java}
private val launcherBackend = new LauncherBackend() {
  override protected def conf: SparkConf = sparkConf  override def onStopRequest(): Unit = {
    if (isClusterMode && appId != null) {
      yarnClient.killApplication(appId)
    } else {
      setState(SparkAppHandle.State.KILLED)
      stop()
    }
  }
}
{code}
Bug: Allow passing V2 functions to data sources via V2 filters
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 08/Sep/21 16:42
Affects Version/s: 3.3.0

The V2 filter API currently only allow {{NamedReference}} in predicates that are pushed down to data sources. It may be beneficial to allow V2 functions in predicates as well so that we can implement function pushdown. This feature is also supported by Trino (Presto).

One use case is we can pushdown predicates such as {{bucket(col, 32) = 10}} which will allow data sources such as Iceberg to only scan a single partition.
Bug: Fail to load Snappy codec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/21 07:41
Affects Version/s: 3.2.0

snappy-java as a native library should not be relocated in Hadoop shaded client libraries. Currently we use Hadoop shaded client libraries in Spark. If trying to use SnappyCodec to write sequence file, we will encounter the following error:

{code}
[info]   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native Method)                                                                                                 
[info]   at org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)                                                                                                        
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
[info]   at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
[info]   at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
[info]   at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.writeBuffer(SequenceFile.java:1589) 
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.sync(SequenceFile.java:1605)
[info]   at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1629) 
{code}
Bug: Upgrade R version to 4.1.1 in CI images
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/21 01:26
Affects Version/s: 3.3.0

https://developer.r-project.org/#:~:text=Release%20plans,on%202021%2D08%2D10.

R 4.1.1 is released. We might better to test the latest version of R with SparkR.
Bug: Perf issue in SchemaPruning when a struct has many fields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/21 03:11
Affects Version/s: 3.1.2

Seeing a significant performance degradation in query processing when a table contains a significantly large number of fields (>10K).

Here's the stacktraces while processing a query
{code:java}
   java.lang.Thread.State: RUNNABLE   java.lang.Thread.State: RUNNABLE at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:285) at scala.collection.TraversableLike$$Lambda$296/874023329.apply(Unknown Source) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at scala.collection.TraversableLike.map(TraversableLike.scala:285) at scala.collection.TraversableLike.map$(TraversableLike.scala:278) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198) at org.apache.spark.sql.types.StructType.fieldNames(StructType.scala:108) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$.$anonfun$sortLeftFieldsByRight$1(SchemaPruning.scala:70) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$.$anonfun$sortLeftFieldsByRight$1$adapted(SchemaPruning.scala:70) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$$$Lambda$3963/249742655.apply(Unknown Source) at scala.collection.TraversableLike.$anonfun$filterImpl$1(TraversableLike.scala:303) at scala.collection.TraversableLike$$Lambda$403/465534593.apply(Unknown Source) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:302) at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:296) at scala.collection.mutable.ArrayOps$ofRef.filterImpl(ArrayOps.scala:198) at scala.collection.TraversableLike.filter(TraversableLike.scala:394) at scala.collection.TraversableLike.filter$(TraversableLike.scala:394) at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:198) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$.sortLeftFieldsByRight(SchemaPruning.scala:70) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$.$anonfun$sortLeftFieldsByRight$3(SchemaPruning.scala:75) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$$$Lambda$3965/461314749.apply(Unknown Source) {code}
Bug: countDistinct becomes count when converted to SQL string
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 14/Sep/21 20:29
Affects Version/s: 3.1.1

In pyspark=3.1.1

{code:python}
> F.countDistinct(F.col("bar"))
Column<'count(bar)'>{code}
 
In pyspark=2.4

{code:python}
> F.countDistinct(F.col("bar"))
Column<b'count(DISTINCT foo)'>{code}
 
This creates problems when trying to convert the pyspark function into SQL code to compose SQL queries via:
{code:python}
> F.countDistinct(F.col("bar"))._jc.toString()
{code}
Bug: OverwriteByExpression conversion in DataSourceV2Strategy use wrong deleteExpr translation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/21 03:51
Affects Version/s: 3.1.2

spark version release-3.1.2

we develop a hive datasource v2 plugin to support join among multiple hive clusters.
find that there maybe a bug in OverwriteByExpression conversion

code debug at https://github.com/apache/spark/blob/de351e30a90dd988b133b3d00fa6218bfcaba8b8/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala#L216

where wrong param `deletExpr` used, which will result in duplicate filters