av9ash/GitBugs huggingface dataset:
Bug: When the existing field name is a number, an error will be reported when reading the orc file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/21 12:55
Affects Version/s: 3.3.0

You can use the following methods to reproduce the problem:
{quote}val path = "file:///tmp/test_orc"

spark.range(1).withColumnRenamed("id", "100").repartition(1).write.orc(path)

spark.read.orc(path)
{quote}
The error message is like this:
{quote}org.apache.spark.sql.catalyst.parser.ParseException:
 mismatched input '100' expecting {'ADD', 'AFTER'....

== SQL ==
 struct<100:bigint>
 -------^^^
{quote}
The error is actually issued by this line of code:
{quote}CatalystSqlParser.parseDataType("100:bigint")
{quote}
 

The specific background is that spark calls the above code in the process of converting the schema of the orc file into the catalyst schema.
{quote}// code in OrcUtils
 private def toCatalystSchema(schema: TypeDescription): StructType =
Unknown macro: \{  CharVarcharUtils.replaceCharVarcharWithStringInSchema(CatalystSqlParser.parseDataType(schema.toString).asInstanceOf[StructType]) }{quote}
There are two solutions I currently think of:
 # Modify the syntax analysis of SparkSQL to identify this kind of schema
 # The TypeDescription.toString method should add the quote symbol to the numeric column name, because the following syntax is supported:
{quote}CatalystSqlParser.parseDataType("`100`:bigint")
{quote}

But currently TypeDescription does not support changing the UNQUOTED_NAMES variable, should we first submit a pr to the orc project to support the configuration of this variable。

!image-2021-09-03-20-56-28-846.png!

 

How do spark members think about this issue?
Bug: BrotliCodec doesn't support Apple Silicon on MacOS
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/21 02:17
Affects Version/s: 3.3.0

{code}
[info] Caused by: java.lang.UnsatisfiedLinkError: Couldn't load native library 'brotli'. [LoaderResult: os.name="Mac OS X", os.arch="aarch64", os.version="11.5.2", java.vm.name="OpenJDK 64-Bit Server VM", java.vm.version="25.302-b08", java.vm.vendor="Azul Systems, Inc.", alreadyLoaded="null", loadedFromSystemLibraryPath="false", nativeLibName="libbrotli.dylib", temporaryLibFile="/Users/dongjoon/APACHE/spark-merge/target/tmp/brotli8243220902047076449/libbrotli.dylib", libNameWithinClasspath="/lib/darwin-aarch64/libbrotli.dylib", usedThisClassloader="false", usedSystemClassloader="false", java.library.path="/Users/dongjoon/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."]
[info] 	at org.meteogroup.jbrotli.libloader.BrotliLibraryLoader.loadBrotli(BrotliLibraryLoader.java:35)
[info] 	at org.apache.hadoop.io.compress.BrotliCodec.<init>(BrotliCodec.java:40)
{code}
Bug: Reuse compatible executors for stage-level scheduling
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Sep/21 02:12
Affects Version/s: 3.1.2

Current stage-level scheduling allocated separated set of executors for different profiles. This approach simplified implementation, however is a waste of executor resources when the existing executors have enough resources to run the following tasks.

We proposed to reuse executors by defining a "compatible" executor concept: two executors binding to different resource profiles are compatible only when the executorResources (cores in particular if not defining custom resources) are the same, but taskResources can be different.  When the executors are compatible, the tasks can be allocated to any of them even when in the different profiles.  Users defining profiles should make sure the different taskResources are properly specified against the same executorResources.

The typical user scenario is for different stages, user wants to use different core number for the task with same executor resources. For instance in CPU machine learning scenario, to achieve the best performance, given the same executor resources, when in ETL stage, user will allocate 1 core per task and many tasks, and in the following CPU training stage, user will use more cores per task and less tasks. Reusing executors allows better CPU resource utilization and better performance.

The first PR will focus on reusing executors with same cores without custom resources. A SparkConf option is defined to change the default behavior which is not reusing executors.
Bug: Handle DNS failures in Executor without causing FetchFailed
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Sep/21 17:57
Affects Version/s: 2.3.0

DNS failures caused on the executor nodes causing shuffle nodes to be added to the exclude list. We should handle this failure and not cause a FetchFailed exception in such cases.

This helps in two things : 

1. Stage won't be resubmitted due to FetchFailedException
2. Spark's exclude listing also won't exclude the shuffle service node when the problem is
indeed with the current executor host.
Bug: Add bitwise functions for the BINARY data type
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Sep/21 20:25
Affects Version/s: 3.3.0

Add four new SQL functions operating on the `BINARY` data type for performing bitwise operations: BITAND, BITOR, BITXOR, and BITNOT.

The BITAND, BITOR, and BITXOR functions take two byte strings as input and return the bitwise AND, OR, or XOR of the two input byte strings. The byte size of the result is the maximum of the byte sizes of the inputs, while the result is computed by aligning the two strings with respect to their least significant byte, and left-padding with zeros the shorter of the two inputs.

The BITNOT function is a unary function and returns the input byte string with all the bits negated.
Bug: Support Trigger.AvailableNow on Kafka data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/21 06:08
Affects Version/s: 3.3.0

SPARK-36533 introduces a new trigger Trigger.AvailableNow, but only introduces the new functionality to the file stream source. Given that Kafka data source is the one of major data sources being used in streaming query, we should make Kafka data source support this.
Bug: Remove lz4 hadoop wrapper classes after Hadoop 3.3.2
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 06/Sep/21 21:12
Affects Version/s: 3.3.0

Lz4-java as provided dependency is not correctly excluded from relocation in Hadoop shaded client libraries in Hadoop 3.3.1. (HADOOP-17891)

 

In order to deal the issue without reverting back to non-shade client libraries, we add a few Lz4 Hadoop wrapper classes `LZ4Factory`, `LZ4Compressor`, and `LZ4SafeDecompressor`, under the package `org.apache.hadoop.shaded.net.jpountz.lz4`.

We should remove these wrapper classes after Hadoop 3.3.2 release which should include the fix.
Bug: Cannot resolve attribute with table reference
Status: Open | Priority: Major | Resolution: 
Created: 15/Sep/21 16:06
Affects Version/s: 2.4.7

Spark seems in some cases unable to resolve attributes that contain multi-part names where the first parts reference a table. Here's a repro:
{code:python}
>>> spark.range(3).toDF("col").write.parquet("testdata")

# Single name part attribute is fine
>>> spark.sql("SELECT col FROM parquet.testdata").show()
+---+
|col|
+---+
|  1|
|  0|
|  2|
+---+

# Name part with the table reference fails
>>> spark.sql("SELECT parquet.testdata.col FROM parquet.testdata").show()

AnalysisException: cannot resolve '`parquet.testdata.col`' given input columns: [col]; line 1 pos 7;
'Project ['parquet.testdata.col]
+- Relation[col#50L] parquet
{code}

The expected behavior is that {{parquet.testdata.col}} is recognized as referring to attribute {{col}} in {{parquet.testdata}} (you'd expect {{AttributeSeq.resolve}} matches [this case|https://github.com/apache/spark/blob/b665782f0d3729928be4ca897ec2eb990b714879/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala#L214-L239]).
Bug: Upgrade Kafka to 2.8.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/21 13:31
Affects Version/s: 3.2.1

A few hours ago, Kafka 2.8.1 was released, which includes a bunch of bug fix.
https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html
Bug: Add more Not operator optimizations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/21 22:49
Affects Version/s: 3.3.0

{{BooleanSimplification should be able to do more simplifications for Not operators applying following rules}}
 # {{Not(null) == null}}
 ## {{e.g. IsNull(Not(...)) can be IsNull(...)}}
 # {{(Not(a) = b) == (a = Not(b))}}
 ## {{e.g. Not(...) = true can be (...) = false}}
 # {{(a != b) == (a = Not(b))}}
 ## {{e.g. (...) != true can be (...) = false}}
Bug: Deadlock between map-output-dispatcher and dispatcher-BlockManagerMaster upon migrating shuffle blocks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/21 14:03
Affects Version/s: 3.1.0

I can observe a deadlock on the driver that can be triggered rather reliably in a job with a larger amount of tasks - upon using
{code:java}
spark.decommission.enabled: true
spark.storage.decommission.rddBlocks.enabled: true
spark.storage.decommission.shuffleBlocks.enabled: true
spark.storage.decommission.enabled: true{code}
 

It origins in the {{dispatcher-BlockManagerMaster}} making a call to {{updateBlockInfo}} when shuffles are migrated. This is not performed by a thread from the pool but instead by the {{dispatcher-BlockManagerMaster}} itself. I suppose this was done under the assumption that this would be very fast. However if the block that is updated is a shuffle index block it calls
{code:java}
mapOutputTracker.updateMapOutput(shuffleId, mapId, blockManagerId){code}
for which it waits to acquire a write lock as part of the {{MapOutputTracker}}.

If the timing is bad then one of the {{map-output-dispatchers}} are holding this lock as part of e.g. {{serializedMapStatus}}. In this function {{MapOutputTracker.serializeOutputStatuses}} is called and as part of that we do
{code:java}
if (arrSize >= minBroadcastSize) {
 // Use broadcast instead.
 // Important arr(0) is the tag == DIRECT, ignore that while deserializing !
 // arr is a nested Array so that it can handle over 2GB serialized data
 val arr = chunkedByteBuf.getChunks().map(_.array())
 val bcast = broadcastManager.newBroadcast(arr, isLocal){code}
which makes an RPC call to {{dispatcher-BlockManagerMaster}}. That one however is unable to answer as it is blocked while waiting on the aforementioned lock. Hence the deadlock. The ingredients of this deadlock are therefore: sufficient size of the array to go the broadcast-path, as well as timing of incoming {{updateBlockInfo}} call as happens regularly during decommissioning. Potentially earlier versions than 3.1.0 are affected but I could not sufficiently conclude that.

I have a stacktrace of all driver threads showing the deadlock: [^spark_stacktrace_deadlock.txt]

A coworker of mine wrote a patch that replicates the issue as a test case as well: [^0001-Add-test-showing-that-decommission-might-deadlock.patch]
Bug: Simplify boolean equalities if one side is literal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/21 22:10
Affects Version/s: 3.3.0

The following query does not push down the filter 

```

SELECT * FROM t WHERE (a AND b) = true

```

although the following equivalent query pushes down the filter as expected.

```

SELECT * FROM t WHERE (a AND b) 

```
Bug: Use default WebsocketPingInterval for Kubernetes watches
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/21 16:40
Affects Version/s: 3.3.0

If you access the Kubernetes API via a load balancer (e.g. HAProxy) and have set a tunnel timeout, the following error message is thrown exactly after each timeout.
{code}
>>> 21/09/27 15:35:19 WARN WatchConnectionManager: Exec Failure
java.io.EOFException
        at okio.RealBufferedSource.require(RealBufferedSource.java:61)
        at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
        at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
        at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
        at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{code}

This exception is quite annoying when working interactively with a paused pySpark shell where the driver component runs locally but the executors run in Kubernetes.
Bug: Pull out ordering expressions
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Sep/21 06:26
Affects Version/s: 3.3.0

Similar to [PullOutGroupingExpressions|https://github.com/apache/spark/blob/7fd3f8f9ec55b364525407213ba1c631705686c5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala#L48]. We can pull out ordering expressions to improve order performance. For example:
{code:scala}
sql("create table t1(a int, b int) using parquet")
sql("insert into t1 values (1, 2)")
sql("insert into t1 values (3, 4)")
sql("select * from t1 order by a - b").explain
{code}
{noformat}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Sort [(a#12 - b#13) ASC NULLS FIRST], true, 0
   +- Exchange rangepartitioning((a#12 - b#13) ASC NULLS FIRST, 5), ENSURE_REQUIREMENTS, [id=#39]
      +- FileScan parquet default.t1[a#12,b#13]
{noformat}
The {{Subtract}} will be evaluated 4 times.
Bug: Uniformly execute the mergeContinuousShuffleBlockIdsIfNeeded method first
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 26/Sep/21 06:50
Affects Version/s: 3.1.0

In the ShuffleBlockFetcherIterator.partitionBlocksByFetchMode method, both local and host-local execute the mergeContinuousShuffleBlockIdsIfNeeded method first, but remote blocks executes the method many times in the createFetchRequests method. Can the method of merge blocks be executed only once in advance?
Bug: Misleading Error Message with Invalid Column and Group By
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/21 19:52
Affects Version/s: 3.1.2

When you run a query with an invalid column that also does a group by on a constructed column, the error message you get back references a missing column for the group by rather than the invalid column.

You can reproduce this in pyspark in 3.1.2 with the following code:
{code:python}
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Group By Issue").getOrCreate()
data = spark.createDataFrame(
    [("2021-09-15", 1), ("2021-09-16", 2), ("2021-09-17", 10), ("2021-09-18", 25), ("2021-09-19", 500), ("2021-09-20", 50), ("2021-09-21", 100)],
    schema=["d", "v"]
    )
data.createOrReplaceTempView("data")
# This is valid
spark.sql("select sum(v) as value, date(date_trunc('week', d)) as week from data group by week").show()
# This is invalid because val is the wrong variable
spark.sql("select sum(val) as value, date(date_trunc('week', d)) as week from data group by week").show()
{code}

The error message for the second spark.sql line is
{quote}
pyspark.sql.utils.AnalysisException: cannot resolve '`week`' given input columns: [data.d, data.v]; line 1 pos 81;
'Aggregate ['week], ['sum('val) AS value#21, cast(date_trunc(week, cast(d#0 as timestamp), Some(America/New_York)) as date) AS week#22]
+- SubqueryAlias data
   +- LogicalRDD [d#0, v#1L], false
{quote}
but the actual problem is that I used the wrong variable name in a different part of the query. Nothing is wrong with {{week}} in this case.
Bug: Support sql overwrite a path that is also being read from when partitionOverwriteMode is dynamic
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Sep/21 09:00
Affects Version/s: 3.1.2

{code:java}
// non-partitioned table overwrite
CREATE TABLE tbl (col1 INT, col2 STRING) USING PARQUET;
INSERT OVERWRITE TABLE tbl SELECT 0,1;
INSERT OVERWRITE TABLE tbl SELECT * FROM tbl;

// partitioned table static overwrite
CREATE TABLE tbl (col1 INT, col2 STRING) USING PARQUET PARTITIONED BY (pt1 INT);
INSERT OVERWRITE TABLE tbl PARTITION(p1=2021) SELECT 0 AS col1,1 AS col2;
INSERT OVERWRITE TABLE tbl PARTITION(p1=2021) SELECT col1, col2 FROM WHERE p1=2021;

{code}
When we run the above query, an error will be throwed "Cannot overwrite a path that is also being read from"

We need to support this operation when the spark.sql.sources.partitionOverwriteMode is dynamic
Bug: octet_length/bit_length API is not implemented  on Scala/Python/R
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Sep/21 06:54
Affects Version/s: 3.3.0

* octet_length: caliculate the byte length of strings
 * bit_length: caliculate the bit length of strings

Those two string related functions are only implemented on SparkSQL, not on Scala, Python and R.

Those functions would be useful for multi-bytes character users, who mainly working with those languages.
Bug: Refactor SpecificParquetRecordReaderBase and add more coverage on vectorized Parquet decoding
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/21 17:58
Affects Version/s: 3.3.0

Add a new test suite to add more coverage for Parquet vectorized decoding, focusing on different combinations of Parquet column index, dictionary, batch size, page size, etc.
Bug: Optimization in PushDownPredicates to push all filters in a single iteration has broken  some optimizations in PruneFilter rule
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 28/Sep/21 17:13
Affects Version/s: 3.1.1

It appears that the optimization in PushDownPredicates rule to try to push all filters in a single pass to reduce iteration has broken the PruneFilter rule to substitute with EmptyRelation when the filter condition is a composite and statically evaluates to false either because one of the non redundant predicate is Literal(false) or all the non redundant predicates are null.

The new PushDownPredicate rule is created by chaining CombineFilters, PushPredicateThroughNonJoin and PushPredicateThroughJoin.

so individual filters will get combined as a single filter while being pushed.

But the PruneFilters rule does not substitute it with empty relation if the filter is composite. It is coded to handle single predicates.

The test is falsely passing as it is testing PushPredicateThroughNonJoin, which does not combine filters. 

While  the actual rule in action has an effect produced by CombineFilters. 

In fact I believe all the places in other tests which are testing individually for PushDownPredicateThroughNonJoin or PushDownPredicateThroughJoin should be corrected ( may be with rule PushPredicates) & re tested.

I will add a bug test & open PR.
Bug: Upgrade Scala to 2.12.15
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Sep/21 21:32
Affects Version/s: 3.3.0

https://github.com/scala/scala/releases/tag/v2.12.15

Scala 2.12.15 release improves compatibility with JDK 17 and 18:

- Avoids IllegalArgumentException in JDK 17+ for lambda deserialization
- Upgrades to ASM 9.2, for JDK 18 support in optimizer
Bug: Add provided Guava dependency for network-yarn module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/21 06:19
Affects Version/s: 3.2.0

In Spark 3.1 and earlier the network-yarn module implicitly relies on guava from hadoop-client dependency, which was changed by SPARK-33212 where we moved to shaded Hadoop client which no longer expose the transitive guava dependency. This was fine for a while since we were not using {{createDependencyReducedPom}} so the module picks up the transitive dependency from {{spark-network-common}}. However, this got changed by SPARK-36835 when we restored {{createDependencyReducedPom}} and now it is no longer able to find guava classes:
{code}
mvn test -pl common/network-yarn -Phadoop-3.2 -Phive-thriftserver -Pkinesis-asl -Pkubernetes -Pmesos -Pnetlib-lgpl -Pscala-2.12 -Pspark-ganglia-lgpl -Pyarn
...
[INFO] Compiling 1 Java source to /Users/sunchao/git/spark/common/network-yarn/target/scala-2.12/classes ...
[WARNING] [Warn] : bootstrap class path not set in conjunction with -source 8
[ERROR] [Error] /Users/sunchao/git/spark/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:32: package com.google.common.annotations does not exist
[ERROR] [Error] /Users/sunchao/git/spark/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:33: package com.google.common.base does not exist
[ERROR] [Error] /Users/sunchao/git/spark/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:34: package com.google.common.collect does not exist
[ERROR] [Error] /Users/sunchao/git/spark/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:118: cannot find symbol
  symbol:   class VisibleForTesting
  location: class org.apache.spark.network.yarn.YarnShuffleService
{code}
Bug: "sha2" expression with bit_length of 224 returns incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/21 23:31
Affects Version/s: 2.4.0

{{sha2(input, bit_length)}} returns incorrect results when {{bit_length == 224}}.

 

This bug seems to have been present since the {{sha2}} expression was introduced in 1.5.0.

 

Repro in spark shell:

{{spark.sql("SELECT sha2('abc', 224)").show()}}

 

Spark currently returns a garbled string, consisting of invalid UTF:

 {{#\t}"4�"�B�w��U�*��你���l��}}

The expected return value is: 

{{23097d223405d8228642a477bda255b32aadbce4bda0b3f7e36c9da7}}

 

This appears to happen because the  {{MessageDigest.digest()}} function appears to return bytes intended to be interpreted as a {{BigInt}} rather than a string. Thus, the output of {{MessageDigest.digest()}} must first be interpreted as a {{BigInt}} and then transformed into a hex string.
Bug: Remove broadcast for InSubqueryExec used in DPP
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/21 17:00
Affects Version/s: 3.3.0

Currently we include a broadcast variable in InSubqueryExec. We use it to hold filtering side query result of DPP. It looks weird because we don't use the result in executors but only need the result in the driver during query planning. We already hold the original result, so basically we hold two copied of query result at this moment.

Another thing related is, in pruningHasBenefit we estimate if DPP pruning has benefit when the join type does not support broadcast. Due to the broadcast variable above, we also check the filtering side against the config autoBroadcastJoinThreshold. The config is not for the purpose and it is not a broadcast join. As the broadcast variable is unnecessary, we can remove this check and leave benefit estimation to overhead and pruning size.
Bug: Explicitly specify error codes when ignoring type hint errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/21 00:00
Affects Version/s: 3.3.0

We use a lot of {{type: ignore}} annotation to ignore type hint errors in pandas-on-Spark.

We should explicitly specify the error codes to make it clear what kind of error is being ignored, then the type hint checker can check more cases.
Bug: Union should resolve nested columns as top-level columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/21 07:47
Affects Version/s: 3.3.0

Union, by definition, resolves columns by position. Currently we only follow this behavior at top-level columns, but not nested columns.

As we are making nested columns as first-class citizen, the nested-column-only limitation and the difference between top-level column and nested column do not make sense.