av9ash/GitBugs huggingface dataset:
Bug: complete documenting table_valued function 
Status: In Progress | Priority: Minor | Resolution: 
Created: 06/Aug/20 00:58
Affects Version/s: 3.0.1

In Spark SQL, we have two types of built-in table-valued functions. 1) Can be specified in the FROM clause. For example range; 2) Can be used in SELECT. For example, explode and so on. Currently we only document the 1st type. Need to document the 2nd type too.
Bug: How to disable port 8080 in Spark?
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 04/Aug/20 08:43
Affects Version/s: 2.3.3

I am running Apache Spark 2.3.3 in Standalone Mode with client deploy mode. By default when we start master and start slave, port 8080 and 8081 will open automatically from SPARK application.
Due to security constraint, I would like to disable Spark web UI for master (8080) and all workers (8081).
Bug: Getting local shuffle block clutters the executor logs
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 20/Aug/20 06:05
Affects Version/s: 3.1.0

The below log statement in {{BlockManager.getLocalBlockData}} should be at debug level
{code:java}
logInfo(s"Getting local shuffle block ${blockId}")
{code}
Currently, the executor logs get cluttered with this
{code:java}
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6103_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6132_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6137_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6312_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6323_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6402_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6413_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6694_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6709_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6753_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6822_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6894_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6913_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7052_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7073_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7167_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7194_4964
{code}

This was added with SPARK-20629.
cc. [~holden]
Bug: Fix the config value for shuffled hash join in test in-joins.sql
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/Aug/20 02:12
Affects Version/s: 3.1.0

The config value for enabling shuffled hash join is wrong in test [https://github.com/apache/spark/blob/master/sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/in-joins.sql#L8-L10], which should not be spark.sql.autoBroadcastJoinThreshold=-1. Created this Jira to fix this. This is a very minor change.
Bug: Remove arrow::as_tibble usage in SparkR
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 05/Aug/20 10:30
Affects Version/s: 3.1.0

We increased the minimal version of Arrow R version to 1.0.0 at SPARK-32452, and Arrow R 0.14 dropped {{as_tibble}}. We can remove the usage in SparkR:

{code}
pkg/R/DataFrame.R:                  # Arrow drops `as_tibble` since 0.14.0, see ARROW-5190.
pkg/R/DataFrame.R:                  if (exists("as_tibble", envir = asNamespace("arrow"))) {
pkg/R/DataFrame.R:                    as.data.frame(arrow::as_tibble(arrowTable), stringsAsFactors = stringsAsFactors)
pkg/R/deserialize.R:    # Arrow drops `as_tibble` since 0.14.0, see ARROW-5190.
pkg/R/deserialize.R:    useAsTibble <- exists("as_tibble", envir = asNamespace("arrow"))
pkg/R/deserialize.R:      as_tibble <- get("as_tibble", envir = asNamespace("arrow"))
pkg/R/deserialize.R:      lapply(batches, function(batch) as.data.frame(as_tibble(batch)))
{code}
Bug: Refactor common partitioning/distribution logic to BaseAggregateExec
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 30/Aug/20 07:27
Affects Version/s: 3.1.0

For all three different aggregate physical operator: `HashAggregateExec`, `ObjectHashAggregateExec` and `SortAggregateExec`, they have same `outputPartitioning` and `requiredChildDistribution` logic. Refactor these same logic into their super class `BaseAggregateExec` to avoid code duplication and future bugs (similar to `HashJoin` and `ShuffledJoin`).
Bug: Record metrics of extra BitSet/HashSet in full outer shuffled hash join
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/Aug/20 18:14
Affects Version/s: 3.1.0

This is a followup coming from full outer shuffled hash join PR, where we want to record metrics for size of extra BitSet/HashSet used in full outer shuffled hash join, for matched build side rows. See detailed discussion in [https://github.com/apache/spark/pull/29342/files#r470948744] .
Bug: Optimize BHJ/SHJ inner and semi join with empty hashed relation
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 18/Aug/20 06:11
Affects Version/s: 3.1.0

With `EmptyHashedRelation` introduced in [https://github.com/apache/spark/pull/29389], it inspired me that there's a minor optimization we can apply to broadcast hash join and shuffled hash join if build side hashed relation is empty.

If build side hashed relation is empty (i.e. build side is empty)

1.inner join: we don't need to execute stream side at all, just return an empty iterator - [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala#L152]

2.semi join: we don't need to execute stream side at all, just return an empty iterator - [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala#L227] .

This is not common that build side is empty, but in case it is, we can leverage it to not execute stream side at all for better query CPU/IO performance.
Bug: update duration property for live ui application list and application apis
Status: In Progress | Priority: Trivial | Resolution: 
Created: 10/Aug/20 12:19
Affects Version/s: 3.1.0

"duration" property in response from application list and application APIs of live UI is always "0". we want to let these two APIs return correct value, same with "*Total Uptime*" in live UI's job page
Bug: dataset.sort.select.write.partitionBy sorts wrong column
Status: Closed | Priority: Blocker | Resolution: Not A Problem
Created: 23/Jul/23 06:05
Affects Version/s: 3.4.1

(In this example the dataset is of type Tuple3, and the columns are named _1, _2 and _3)

 

I found -then when AQE is enabled,- that the following code does not produce sorted output (.drop() also have the same problem), unless spark.sql.optimizer.plannedWrite.enabled is set to false.

After further investigation, spark actually sorted wrong column in the following code.

{{dataset.sort("_1")}}
{{.select("_2", "_3")}}
{{.write()}}
{{.partitionBy("_2")}}
{{.text("output");}}

 
(the following workaround is no longer necessary)
-However, if I insert an identity mapper between select and write, the output would be sorted as expected.-

-{{dataset = dataset.sort("_1")}}-
-{{.select("_2", "_3");}}-
-{{dataset.map((MapFunction<Row, Row>) row -> row, dataset.encoder())}}-
-{{.write()}}-
-{{{}.{}}}{{{}partitionBy("_2"){}}}-
-{{.text("output")}}-

Below is the complete code that reproduces the problem.
Bug: Fix the session passed to foreachBatch. 
Status: Reopened | Priority: Critical | Resolution: 
Created: 17/Jul/23 19:20
Affects Version/s: 3.4.1

foreachBatch() in Connect uses initial session used while starting the streaming query. But streaming query uses a cloned session, not the the original session. We should set up the mapping for the cloned session and pass that in. Look for this ticket ID in the code for more context inline. 

 

Another issue with not creating new session id: foreachBatch worker keeps the session alive. The session mapping at Connect server does not expire and query keeps running even if the original client disappears. This keeps the query running.
Bug: Migrated shuffle blocks are encrypted multiple times when io.encryption is enabled 
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 28/Jul/23 15:55
Affects Version/s: 3.1.0

Shuffle blocks upon migration are wrapped for encryption again when being written out to a file on the receiver side.

 

Pull request to fix this: https://github.com/apache/spark/pull/42214

 

Details:

Sender/Read side:

BlockManagerDecommissioner:run()
    blocks = bm.migratableResolver.getMigrationBlocks()
        *dataFile = IndexShuffleBlockResolver:getDataFile(...)*
       buffer = FileSegmentManagedBuffer(..., dataFile)
                       *^ This reads straight from disk without decryption*
    blocks.foreach((blockId, buffer) => bm.blockTransferService.uploadBlockSync(..., buffer, ...))
        -> uploadBlockSync() -> uploadBlock(..., buffer, ...)
            -> client.uploadStream(UploadBlockStream, buffer, ...)
 - Notice that there is no decryption here on the sender/read side.

Receiver/Write side:

NettyBlockRpcServer:receiveStream() <--- This is the UploadBlockStream handler
    putBlockDataAsStream()
        migratableResolver.putShuffleBlockAsStream()
            *-> file = IndexShuffleBlockResolver:getDataFile(...)*
            -> tmpFile = (file + .<uuid> extension)
            *-> Creates an encrypting writable channel to a tmpFile using serializerManager.wrapStream()*
            -> onData() writes the data into the channel
            -> onComplete() renames the tmpFile to the file
 - Notice:

 * Both getMigrationBlocks()[read] and putShuffleBlockAsStream()[write] target IndexShuffleBlockResolver:getDataFile()
 * The read path does not decrypt but the write path encrypts.
 * As a thought exercise: if this cycle happens more than once (where this receiver is now a sender) even if we assume that the shuffle blocks are initially unencrypted*, then bytes in the file will just have more and more layers of encryption applied to it each time it gets migrated.
 * *In practice, the shuffle blocks are encrypted on disk to begin with, this is just a thought exercise
Bug: Path Inconsistency when Operating statCache within Yarn Client
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 02/Jul/23 15:17
Affects Version/s: 3.4.0

The *addResource* from *ClientDistributedCacheManager* can add *FileStatus* to 

*statCache* when it is not yet cached. Also, there is a subtle bug from *isPublic* from 

*getVisibility* method. *uri.getPath()* will not retain URI information like 

scheme, host, etc. So, the *uri* passed to checkPermissionOfOther will differ from the original {*}uri{*}.

For example, if uri is "file:/foo.invalid.com:8080/tmp/testing", then 
{code:java}
uri.getPath -> /foo.invalid.com:8080/tmp/testing
uri.toString -> file:/foo.invalid.com:8080/tmp/testing{code}
The consequence of this bug is that we will *double RPC calls* when the resources are remote, which is unnecessary. We see nontrivial overhead when checking those resources from our HDFS, especially when HDFS is overloaded. 

 

Ref: related code within *ClientDistributedCacheManager*
{code:java}
def addResource(...) {
    val destStatus = statCache.getOrElse(destPath.toUri(), fs.getFileStatus(destPath))
val visibility = getVisibility(conf, destPath.toUri(), statCache)
}
private[yarn] def getVisibility() {
isPublic(conf, uri, statCache)
}
private def isPublic(conf: Configuration, uri: URI, statCache: Map[URI, FileStatus]): Boolean = {
val current = new Path(uri.getPath()) // Should not use getPath
checkPermissionOfOther(fs, uri, FsAction.READ, statCache)
}
{code}
Bug: Spark Driver and executors are not transferring from standby master to Alive Master.
Status: Open | Priority: Critical | Resolution: 
Created: 14/Jul/23 05:24
Affects Version/s: 3.3.2

Issue with the Spark driver and executors transferring from Standby Master to Alive Master.

Spark is deployed with Zookeeper as cluster manager. Ideally if standby node(server) goes down it should transfer applications running on standby node to alive node.
Application transformations are killed in the Standby Master.
  # Just consider we have two Application drivers were running on the Standby Master server.
 # If we forcefully shutdown, This should trigger the failover scenario and application should move to Alive node. But, The Standby Master node both the application drivers were not transferring to Active master instead those were killing.

Check attachment for reference.
 
Steps to reproduce
 * Deploy Spark HA cluster with Zookeeper
 * Deploy multiple application until we have drivers and executor assigned on standby node.
 * Now reboot the server
 * Check if all the applications were moving to Active Master.
Bug: Introduce simpe conf system for sql/api
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jul/23 22:41
Affects Version/s: 3.4.1

Create a simple conf system for classes in sql/api. This is needed for a number of classes that are moved from sql/catalyst to sql/api that require configuration access (e.g. timeZone, parsing behavior, ...).

The change will add a small common interface that allows you to read the needed configurations, this interface is implemented by SQLConf and SQLConf will be used when we are executing on the driver, and there will be an implementation using the default values for when we are in Connect mode.
Bug: Overwriting the same partition of a partitioned table multiple times with empty data yields non-idempotent results
Status: Open | Priority: Major | Resolution: 
Created: 18/Jul/23 08:03
Affects Version/s: 3.1.3

 

Preparation:
Create a simple partition table using spark version 3.x, for example:

 
{code:java}
spark-sql> create table test1 (a int) partitioned by (dt string);
Time taken: 0.219 seconds{code}
 

 
 * Overwrite a new partition with empty data, and you can see that the partition information and the corresponding HDFS path are generated , for example:

{code:java}

spark-sql> insert overwrite table test1 partition(dt='20230702') select 2 where 1 <> 1;
Time taken: 0.992 seconds
spark-sql> dfs -ls /user/hive/warehouse/test1;
Found 2 items
-rw-r--r-- 2 hadoop hadoop 0 2023-07-18 14:41 /user/hive/warehouse/test1/_SUCCESS
drwxrwxrwx- hadoop hadoop 0 2023-07-18 14:41 /user/hive/warehouse/test1/dt=20230702
spark-sql> show partitions test1;
dt=20230702
Time taken: 0.162 seconds, Fetched 1 row(s)
{code}
 * When re-running the insert overwrite statement, you can see that the HDFS path corresponding to this partition does not exist.

 
{code:java}
spark-sql> insert overwrite table test1 partition(dt='20230702') select 2 where 1 <> 1;
Time taken: 0.706 seconds
spark-sql> dfs -ls /user/hive/warehouse/test1;
Found 1 items
-rw-r--r--   2 hadoop hadoop          0 2023-07-18 14:45 /user/hive/warehouse/test1/_SUCCESS
spark-sql> show partitions test1;
dt=20230702
Time taken: 0.183 seconds, Fetched 1 row(s){code}
For subsequent tasks that need to use this HDFS path, an exception that the path does not exist will be thrown, which caused us trouble.

 

I was expecting to execute the same statement multiple times to get the same result, {*}not non-idempotent{*}. thanks.
Bug: Remove/Reduce usage of TypeTag in public APIs
Status: Open | Priority: Major | Resolution: 
Created: 13/Jul/23 06:18
Affects Version/s: 4.0.0

In scala 3 there is no replacement for TypeTag ([https://github.com/lampepfl/dotty-feature-requests/issues/62)] it seems like the intended replacement is just to do compile time reflection using macros.

In order to be able to make a scala 3 compatible release of spark 4 we should reduce/remove usage of TypeTag in public APis like `org.apache.spark.sql.functions.udf` instead I suggest that it would be better it just required a `ExpressionEncoder` (or some other spark owned type class) directly. That why we would just need to provide a scala 3 specific way only for the type class derivation and not for the other APIs that are currently using TypeTag.
Bug: Use the latest minikube in K8s IT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/23 08:31
Affects Version/s: 3.4.2

*BEFORE*

* Minikube: v1.30.1
* K8s: v1.26.3

*AFTER*

* Minikube: v1.32.0
* K8s: v1.28.3
Bug: Eliminate the Join by combine multiple Aggregates
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/23 11:09
Affects Version/s: 3.5.0

Recently, I investigate the test case q28 which is belong to the TPC-DS queries.

The query contains multiple scalar subquery with aggregation and connected with inner join.
If we can merge the filters and aggregates, we can scan data source only once and eliminate the join so as avoid shuffle. Obviously, this change will improve the performance.
Bug: ExecutorPodsAllocator doesn't create new executors if no pod snapshot captured pod creation
Status: Open | Priority: Major | Resolution: 
Created: 31/Jul/23 13:01
Affects Version/s: 3.4.1

There’s a following race condition in ExecutorPodsAllocator when running a spark application with static allocation on kubernetes with numExecutors >= 1:
 * Driver requests an executor
 * exec-1 gets created and registers with driver
 * exec-1 is moved from {{newlyCreatedExecutors}} to {{schedulerKnownNewlyCreatedExecs}}
 * exec-1 got deleted very quickly (~1-30 sec) after registration
 * {{ExecutorPodsWatchSnapshotSource}} fails to catch the creation of the pod (e.g. websocket connection was reset, k8s-apiserver was down, etc.)
 * {{ExecutorPodsPollingSnapshotSource}} fails to catch the creation because it runs every 30 secs, but executor was removed much quicker after creation
 * exec-1 is never removed from {{schedulerKnownNewlyCreatedExecs}}
 * {{ExecutorPodsAllocator}} will never request new executor because it’s slot is occupied by exec-1, due to {{schedulerKnownNewlyCreatedExecs}} never being cleared.

 

Put up a fix here https://github.com/apache/spark/pull/42297
Bug: Drop mesos support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/23 06:08
Affects Version/s: 4.0.0

[https://spark.apache.org/docs/latest/running-on-mesos.html]

 

{_}Note{_}: Apache Mesos support is deprecated as of Apache Spark 3.2.0. It will be removed in a future version.
Bug: Completely make hive as a data source
Status: Open | Priority: Major | Resolution: 
Created: 24/Jul/23 04:25
Affects Version/s: 3.5.0

Now, hive is a different data source from other data sources. In Spark Project, Hive have many special logic and burden the cost of maintenance . Like presto, hive is only a connector. Is it possible that we can  make hive as a data source completely?

Surely, I know that it's very difficult. It has many historical problems and compatible problems. Could we reduce these problems as possible as we can if we release 4.0?

I just wanna start a discussion to collect more people's suggestion. Any suggestion is welcome. I just feel 4.0 is a good opportunity to discuss this issue.

If I am wrong, it's welcome to point it out.
Bug: Add convertJavaTimestampToTimestamp in JDBCDialect API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jul/23 18:35
Affects Version/s: 3.5.0

A new method, {{{}convertJavaTimestampToTimestamp{}}}, is introduced to the JDBCDialects API, providing the capability for JDBC dialects to override the default Java timestamp conversion behavior. This enhancement is particularly beneficial for databases such as PostgreSQL, which feature special values for timestamps representing positive and negative infinity. 

The pre-existing default behavior of timestamp conversion potentially triggers an overflow due to these special values (i.e. The executor would crash if you select a column that contains infinity timestamps in PostgreSQL.) By integrating this new function, we can mitigate such issues, enabling more versatile and robust timestamp value conversions across various JDBC-based connectors.
Bug: Bloom filter is not added for left outer join if the left side table is smaller than broadcast threshold.
Status: Open | Priority: Major | Resolution: 
Created: 05/Jul/23 05:11
Affects Version/s: 3.4.1

In case of left outer join, even if the left side table is small enough to be broadcasted, shuffle join is used. This is because of the property of the left outer join. If the left side is broadcasted in left outer join, the result generated will be wrong. But this is not taken care of in bloom filter. While injecting the bloom filter, if lest side is smaller than broadcast threshold, bloom filter is not added. It assumes that the left side will be broadcast and there is no need for a bloom filter. This causes bloom filter optimization to be missed in case of left outer join with small left side and huge right-side table.
Bug: ExecutorDeadException when using decommissioning without external shuffle service
Status: Open | Priority: Major | Resolution: 
Created: 12/Jul/23 15:32
Affects Version/s: 3.4.0

Hey, we are trying to use executor decommissioning without external shuffle service. We are trying to understand:
 # How often should we expect to see ExecutorDeadException? How is information about changes to location of blocks is propagated?
 # Whether the task should be re-submited if we hit that during decommissioning?

 

Current behavior that we observe:
 # Executor 1 is decommissioned
 # Driver successfully removes executor 1's block manager [here|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala#L44]
 # A task is started on executor 2
 # We hit `ExecutorDeadException` on executor 2 when trying to fetch blocks from executor 1 [here|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala#L139-L140]
 # Task on executor 2 fails
 # Stage fails
 # Stage is re-submitted and succeeds

As far as we understand, this happens because executor 2 has stale [map status cache|https://github.com/apache/spark/blob/87a5442f7ed96b11051d8a9333476d080054e5a0/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L1235-L1236]

Is that expected behavior? Shouldn't the task be retried in that case instead of whole stage failing and being retried? This makes Spark job execution longer, especially if there are a lot of decommission events.

 

Also found [this comment|https://github.palantir.build/foundry/spark/blob/aad028ae02011b079e8812f7e63869323cc1ed78/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L113-L115], which makes sense for FetchFailures w/o decommissioning, but with decommissioning data could have been migrated - and we need to fetch a new location. Maybe it makes sense to special case this codepath to check whether executor was decommissioned? Since https://issues.apache.org/jira/browse/SPARK-40979 we already store that information.
Bug: Improve Scala StreamingQueryListener to provide users a way to access the Spark session for Spark Connect
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Jul/23 22:29
Affects Version/s: 3.5.0

Improve the Listener to provide users a way to access the Spark session and perform arbitrary actions inside the Listener. Right now users can use `val spark = SparkSession.builder.getOrCreate()` to create a Spark session inside the Listener, but this is a legacy session instead of a connect remote session.