av9ash/GitBugs huggingface dataset:
Bug: Fix Docker release tool for branch-3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/22 04:55
Affects Version/s: 3.2.2

The {{do-release-docker.sh}} doesn't work in branch-3.2 now, and gives the following error:
{code:java}
#5 917.4 g++ -std=gnu++14 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o testthat.so init.o reassign.o test-catch.o test-example.o test-runner.o -L/usr/lib/R/lib -lR
#5 917.5 installing to /usr/local/lib/R/site-library/00LOCK-testthat/00new/testthat/libs
#5 917.5 ** R
#5 917.5 ** inst
#5 917.5 ** byte-compile and prepare package for lazy loading
#5 924.4 ** help
#5 924.6 *** installing help indices
#5 924.7 *** copying figures
#5 924.7 ** building package indices
#5 924.9 ** installing vignettes
#5 924.9 ** testing if installed package can be loaded from temporary location
#5 925.1 ** checking absolute paths in shared objects and dynamic libraries
#5 925.1 ** testing if installed package can be loaded from final location
#5 925.5 ** testing if installed package keeps a record of temporary installation path
#5 925.5 * DONE (testthat)
#5 925.8 ERROR: dependency 'pkgdown' is not available for package 'devtools'
#5 925.8 * removing '/usr/local/lib/R/site-library/devtools'
#5 925.8
#5 925.8 The downloaded source packages are in
#5 925.8        '/tmp/Rtmp3nJI60/downloaded_packages'
#5 925.8 Warning messages:
#5 925.8 1: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat",  :
#5 925.8   installation of package 'textshaping' had non-zero exit status
#5 925.8 2: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat",  :
#5 925.8   installation of package 'ragg' had non-zero exit status
#5 925.8 3: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat",  :
#5 925.8   installation of package 'pkgdown' had non-zero exit status
#5 925.8 4: In install.packages(c("curl", "xml2", "httr", "devtools", "testthat",  :
#5 925.8   installation of package 'devtools' had non-zero exit status
#5 926.0 Error in loadNamespace(x) : there is no package called 'devtools'
#5 926.0 Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
#5 926.0 Execution halted {code}
Bug: Do not wrap exceptions thrown in SparkHadoopWriter.write with SparkException
Status: Closed | Priority: Major | Resolution: Won't Fix
Created: 10/Nov/22 13:26
Affects Version/s: 3.4.0

This is similar to https://issues.apache.org/jira/browse/SPARK-40488.

Exceptions thrown in SparkHadoopWriter.write are wrapped with SparkException("Job aborted.").

This wrapping provides little extra information, but generates a long stacktrace, which hinders debugging when error happens.
Bug: Spark Submitter generates a ConfigMap with the same name
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Nov/22 03:12
Affects Version/s: 3.1.2

*Description of the issue:*

There's a problem with submitting spark jobs to K8s cluster: the library generates and reuses the same name for config maps (for drivers and executors). Ideally, for each job 2 config maps should be created: for a driver and an executor. However, the library creates only one driver config map for all jobs (in some cases it generates only one executor map for all jobs in the same manner). So, if I run 5 jobs, then only one driver config map will be generated and used for every job.  During those runs we experience issues when deleting pods from the cluster: executors pods are endlessly created and immediately terminated overloading cluster resources.

 

*The reason of the issue:*

This problem occurs because of the *KubernetesClientUtils* class in which we have *configMapNameExecutor* and *configMapNameDriver* as constants. It seems to be incorrect and should be urgently fixed. I've prepared some changes for review to fix the issue (tested in the cluster of our project).

 

*Steps to reproduce the issue:*

 
 # Create a *KubernetesClientApplication* object.
 # Submit at least 2 jobs (sequentially or using *Thread* for running in parallel).

 

*The results of my observations according to the steps are as follows:*
 # Spark 3.1.2 - The same config map in K8S will be overwritten which means all the jobs will point to the same config map.
 # Spark 3.3.* -  For the first job a new config map will be created. For other jobs an exception will be thrown (the K8S Fabric library does not allow to create a new config map with the existing name).
Bug: Add PartialAggregate and FinalAggregate logic operators
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Nov/22 03:41
Affects Version/s: 3.4.0

To implement these features:
1. Push partial distinct through join: https://github.com/apache/spark/pull/36552/files#diff-49d65c8b39e8895b5916de304972d60dd8651c2bf4eb3a985368b2a4ed278633
2. Partial deduplicate the right side of left semi/anti join: https://github.com/apache/spark/pull/36552/files#diff-adcf229187908c33d4785a96700bd598ccad216bb8bc94f64cf75bda116aa276
3. Push down limit through aggregate if it is group only: https://github.com/wangyum/spark/commit/dac339ac4fee8bfdf66276ef9f998ae1baf47c2e
Bug: Support Bit manipulation function SHIFTLEFT
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/22 00:21
Affects Version/s: 3.1.4

The function shall shift an integer value left by a specified number of bits. As the most significant bit will be taken out of the original value, it shall discard and the least significant bit becomes 0
SHIFTLEFT(integer_type a, INT positions)
Bug: Support Bit manipulation function ROTATERIGHT
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/22 00:19
Affects Version/s: 3.1.4

Support Bit manipulation function ROTATERIGHT

The function shall rotate an integer value right by a specified number of bits. As the least significant bit will be taken out of the original value, if it is a 1 bit, it shall be "rotated" back to the most significant bit. Therefore, the final value will have the same number of 1 bits as the original value, just in different positions
ROTATERIGHT(integer_type a, INT positions)
Bug: Support Bit manipulation function ROTATELEFT
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/22 00:17
Affects Version/s: 3.1.4

Support Bit manipulation function ROTATELEFT

The function shall rotate an integer value left by a specified number of bits. As the most significant bit will be taken out of the original value, if it is a 1 bit, it shall be "rotated" back to the least significant bit.
*ROTATELEFT(integer_type a, INT positions)*
Bug: Rename `MULTI_VALUE_SUBQUERY_ERROR` to `SCALAR_SUBQUERY_TOO_MANY_ROWS `
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/22 03:34
Affects Version/s: 3.4.0

The current error class name "MULTI_VALUE_SUBQUERY_ERROR" is not clearly enough to brief the situation.

`SCALAR_SUBQUERY_TOO_MANY_ROWS` would be more readable since scalar subquery is the industrial term.
Bug: Metaspace OOM when Local run dev/make-distribution.sh 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/22 05:46
Affects Version/s: 3.4.0

Run
{code:java}
dev/make-distribution.sh --tgz -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive  {code}
{code:java}
[ERROR] ## Exception when compiling 19 sources to /Users/yangjie01/SourceCode/git/spark-mine-12/connector/avro/target/scala-2.12/classes
java.lang.OutOfMemoryError: Metaspace
java.lang.ClassLoader.defineClass1(Native Method)
java.lang.ClassLoader.defineClass(ClassLoader.java:757)
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
java.net.URLClassLoader.access$100(URLClassLoader.java:74)
java.net.URLClassLoader$1.run(URLClassLoader.java:369)
java.net.URLClassLoader$1.run(URLClassLoader.java:363)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findClass(URLClassLoader.java:362)
java.lang.ClassLoader.loadClass(ClassLoader.java:419)
scala_maven.ScalaCompilerLoader.loadClass(ScalaCompilerLoader.java:44)
java.lang.ClassLoader.loadClass(ClassLoader.java:352)
scala.collection.immutable.Set$Set2.$plus(Set.scala:170)
scala.collection.immutable.Set$Set2.$plus(Set.scala:164)
scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:28)
scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:24)
scala.collection.TraversableLike$grouper$1$.apply(TraversableLike.scala:467)
scala.collection.TraversableLike$grouper$1$.apply(TraversableLike.scala:455)
scala.collection.mutable.HashMap$$anon$1.$anonfun$foreach$2(HashMap.scala:153)
scala.collection.mutable.HashMap$$anon$1$$Lambda$68804/294594059.apply(Unknown Source)
scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:153)
scala.collection.TraversableLike.groupBy(TraversableLike.scala:524)
scala.collection.TraversableLike.groupBy$(TraversableLike.scala:454)
scala.collection.AbstractTraversable.groupBy(Traversable.scala:108)
scala.tools.nsc.settings.Warnings.$init$(Warnings.scala:91)
scala.tools.nsc.settings.MutableSettings.<init>(MutableSettings.scala:28)
scala.tools.nsc.Settings.<init>(Settings.scala:19)
scala.tools.nsc.Settings.<init>(Settings.scala:20)
xsbt.CachedCompiler0.<init>(CompilerBridge.scala:79) {code}
Bug: Failure of ProtobufCatalystDataConversionSuite.scala
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/22 12:08
Affects Version/s: 3.4.0

To reproduce the issue, set the seed to 38:

{code}
diff --git a/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala b/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala
index 271c5b0fec..080bf1eb1f 100644
--- a/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala
+++ b/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala
@@ -123,7 +123,7 @@ class ProtobufCatalystDataConversionSuite
     StringType -> ("StringMsg", ""))

   testingTypes.foreach { dt =>
-    val seed = 1 + scala.util.Random.nextInt((1024 - 1) + 1)
+    val seed = 38
     test(s"single $dt with seed $seed") {

       val (messageName, defaultValue) = catalystTypesToProtoMessages(dt.fields(0).dataType)
{code}

and run the test:

{code}
build/sbt "test:testOnly *ProtobufCatalystDataConversionSuite"
{code}
which fails with NPE:
{code}
[info] - single StructType(StructField(double_type,DoubleType,true)) with seed 38 *** FAILED *** (10 milliseconds)
[info]   java.lang.NullPointerException:
[info]   at org.apache.spark.sql.protobuf.ProtobufCatalystDataConversionSuite.$anonfun$new$2(ProtobufCatalystDataConversionSuite.scala:134)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
{code}
Bug: Self-union streaming query may fail when using readStream.table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/22 05:40
Affects Version/s: 3.4.0

In a self-union query, the batch plan created by the source will be shared by multiple nodes in the plan. When we transform the plan, the batch plan will be visited multiple times. Hence, the first visit will set the CatalogTable and the second visit will try to set it again and fail the query at [this line|https://github.com/apache/spark/blob/406d0e243cfec9b29df946e1a0e20ed5fe25e152/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L625].
Bug: Pre-compute to eliminate ScalaReflection calls after deserializer is created
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/22 06:40
Affects Version/s: 3.4.0

Currently when `ScalaReflection` returns a deserializer, for a few complex types, such as array, map, udt, etc, it creates functions that may still touch `ScalaReflection` after the deserializer is created.

`ScalaReflection` is a performance bottleneck for multiple threads as it holds multiple global locks. We can refactor `ScalaReflection.deserializerFor` to pre-compute everything that needs to touch `ScalaReflection` before creating the deserializer. After this, once the deserializer is created, it can be reused by multiple threads without touching `ScalaReflection.deserializerFor` any more and it will be much faster.
Bug: Make CommandResult extend Command
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Nov/22 17:32
Affects Version/s: 3.4.0

CommandResult is the logical plan node that stores the results from a command. We want this to still be considered a command, rather than e.g. a query, so we should extend the trait Command which would allow it to pass various checks for commands (such as [this one|https://github.com/apache/spark/blob/f4ff2d16483f7da2c7ab73c7cfec75bb9e91064d/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala#L54-L57]).
Bug: Remove ARM limitation for YuniKorn from docs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Nov/22 01:43
Affects Version/s: 3.3.1

YuniKorn does support ARM. There is no limitation. YuniKorn 1.1 fully supports deploying and running on ARM. Multi architecture images are published as part of each release.

The limitation mentioned in the documentation that YuniKorn does not support ARM is incorrect and should be removed:
https://spark.apache.org/docs/3.3.1/running-on-kubernetes.html#limitations
Bug: Upgrade Apache Ivy to 2.5.1
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Nov/22 17:47
Affects Version/s: 3.4.0

[CVE-2022-37865|https://www.cve.org/CVERecord?id=CVE-2022-37865]
and
[CVE-2022-37866|https://nvd.nist.gov/vuln/detail/CVE-2022-37866]
Bug: Rewrite test RENAME_SRC_PATH_NOT_FOUND to trigger the error from user space
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Nov/22 04:55
Affects Version/s: 3.4.0

Rewrite the test for error class *RENAME_SRC_PATH_NOT_FOUND* in [QueryExecutionErrorsSuite.scala|https://github.com/apache/spark/pull/38782/files/bea17e4fa61a06ff566b0ff1c3fcc39fa1100912#diff-b1989c7e0e4b50291fb7bdd2993da387102c669a47fe6e2077b23b37d78b18be] to trigger the error from user space. The current test uses non-user-facing class FileSystemBasedCheckpointFileManager directly to trigger the error. (see [https://github.com/apache/spark/pull/38782/files/bea17e4fa61a06ff566b0ff1c3fcc39fa1100912#diff-b1989c7e0e4b50291fb7bdd2993da387102c669a47fe6e2077b23b37d78b18beR680] )

Done when: the test uses user-facing APIs as much as possible.

Proposed solution: rewrite the test following the example of [https://github.com/apache/spark/pull/38782/files/bea17e4fa61a06ff566b0ff1c3fcc39fa1100912#diff-b1989c7e0e4b50291fb7bdd2993da387102c669a47fe6e2077b23b37d78b18beR641]

See [https://github.com/apache/spark/pull/38782#discussion_r1033013064] for more context
Bug: Leverage shuffle key as bucketing properties
Status: In Progress | Priority: Minor | Resolution: 
Created: 27/Nov/22 13:38
Affects Version/s: 3.3.1

I'm not sure if I'm not missing anything trivial.

In a typical process, many datasets get materialized and many of them after a shuffle (e.g join). then they would again be involved in further actions and often use the same key.

Wouldn't it make sense to save the shuffle key along with the table to avoid unnecessary shuffles?

Also, the implementation seems quite straightforward - to just leverage the bucketing mechanism.
Bug: Task finished before speculative task scheduled leads to holding idle executors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Nov/22 04:13
Affects Version/s: 3.2.2

When task finished before speculative task has been scheduled by DAGScheduler, then the speculative tasks will be considered as pending and count towards the calculation of number of needed executors, which will lead to request more executors than needed
h2. Background & Reproduce

In one of our production job, we found that ExecutorAllocationManager was holding more executors than needed. 

We found it's difficult to reproduce in the test environment. In order to stably reproduce and debug, we temporarily annotated the scheduling code of speculative tasks in TaskSetManager:363 to ensure that the task be completed before the speculative task being scheduled.
{code:java}
// Original code
private def dequeueTask(
    execId: String,
    host: String,
    maxLocality: TaskLocality.Value): Option[(Int, TaskLocality.Value, Boolean)] = {
  // Tries to schedule a regular task first; if it returns None, then schedules
  // a speculative task
  dequeueTaskHelper(execId, host, maxLocality, false).orElse(
    dequeueTaskHelper(execId, host, maxLocality, true))
} 
// Speculative task will never be scheduled
private def dequeueTask(
    execId: String,
    host: String,
    maxLocality: TaskLocality.Value): Option[(Int, TaskLocality.Value, Boolean)] = {
  // Tries to schedule a regular task first; if it returns None, then schedules
  // a speculative task
  dequeueTaskHelper(execId, host, maxLocality, false)
}  {code}
Referring to examples in SPARK-30511

You will see when running the last task, we would be hold 38 executors (see attachment), which is exactly (149 + 1) / 4 = 38. But actually there are only 2 tasks in running, which requires Math.min(20, 2/4) = 20 executors indeed.
{code:java}
./bin/spark-shell --master yarn --conf spark.speculation=true --conf spark.executor.cores=4 --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.minExecutors=20 --conf spark.dynamicAllocation.maxExecutors=1000 {code}
{code:java}
val n = 4000
val someRDD = sc.parallelize(1 to n, n)
someRDD.mapPartitionsWithIndex( (index: Int, it: Iterator[Int]) => {
if (index > 3998) {
    Thread.sleep(1000 * 1000)
} else if (index > 3850) {
    Thread.sleep(50 * 1000) // Fake running tasks
} else {
    Thread.sleep(100)
}
Array.fill[Int](1)(1).iterator{code}
 

I will have a PR ready to fix this issue
Bug: protoc-3.21.9-linux-x86_64.exe requires GLIBC_2.14
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Nov/22 11:15
Affects Version/s: 3.4.0

Test compile Protobuf module on CentOS release 6.3, the default glibc version is 2.12, this will cause compilation to fail as follows:
{code:java}
[ERROR] /home/disk1/spark-ut/spark/connector/protobuf/src/test/resources/protobuf/timestamp.proto [0:0]: /tmp/protoc6599263403262688374.exe: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/protoc6599263403262688374.exe)
[ERROR] /home/disk1/spark-ut/spark/connector/protobuf/src/test/resources/protobuf/timestamp.proto [0:0]: /tmp/protoc6599263403262688374.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /tmp/protoc6599263403262688374.exe)
[ERROR] /home/disk1/spark-ut/spark/connector/protobuf/src/test/resources/protobuf/timestamp.proto [0:0]: /tmp/protoc6599263403262688374.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /tmp/protoc6599263403262688374.exe)
[ERROR] /home/disk1/spark-ut/spark/connector/protobuf/src/test/resources/protobuf/timestamp.proto [0:0]: /tmp/protoc6599263403262688374.exe: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /tmp/protoc6599263403262688374.exe){code}
Bug: Identical expressions should not be used on both sides of a binary operator
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 04/Nov/22 13:39
Affects Version/s: 3.4.0

Identical expressions should not be used on both sides of a binary operator.
There are a lot of this 278 total,

{code:java}
self.check_extension(pser != pser, psser != psser)
{code}


in pandas API test code. 

[Issues - spark-python - Bjørn Jørgensen (sonarcloud.io)|https://sonarcloud.io/project/issues?languages=py&resolved=false&rules=python%3AS1764&id=spark-python]

We either need to fix this or remove them.
Bug: Can't build Spark docker image with Java 17
Status: Closed | Priority: Minor | Resolution: Invalid
Created: 09/Nov/22 17:10
Affects Version/s: 3.3.1

I'm trying to build Spark docker image with Java 17 according to usage description from {{{}docker-image-tool.sh{}}}:
{code:java}
 ./bin/docker-image-tool.sh -r $REGISTRY -t ${VERSION} -f kubernetes/dockerfiles/spark/Dockerfile.java17 build

{code}
But I get the following error:
{code:java}
./bin/docker-image-tool.sh: line 74: cd: kubernetes/dockerfiles/spark: No such file or directory

{code}
As far as I understand the script verifies the argument for presence before it creates a build context and copies content from {{{}resource-managers/kubernetes/docker/src/main/dockerfiles{}}}.
Bug: Add code example for JDBC data source with partitionColumn
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Nov/22 09:14
Affects Version/s: 3.4.0

We should add code example for JDBC data source with partitionColumn in our documentation - [https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html,] to better guide users.
Bug: Spark Web UI 
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 07/Nov/22 19:47
Affects Version/s: 3.2.1

Hi everyone,

I have the following setup. 
 * Spark 3.2.1 created from bitnami images
 * Forwarded port 8080 to 12345 in Docker container with spark master: 0.0.0.0:12345->8080/tcp 

In spark-config:
 * spark.ui.reverseProxy=true
 * spark.ui.reverseProxyUrl=[https://host.com/project/spark/]

 

Apache 2.4.6 (same host):

 
{code:java}
 RewriteCond %{HTTP:Connection} Upgrade [NC]
          RewriteCond %{HTTP:Upgrade} websocket [NC]
          RewriteRule /project/spark/(.*) ws://host.com:12345/$1 [NE,P,L]          ProxyPass /project/spark/ http://host.com:10721/
          ProxyPassReverse /project/spark/ http://host.com:12345/
 {code}
 

 

 

When I access [http://host.com/project/spark/] master ui seems working fine. However If I click on application UI  (running app)

!spark.png!

 

Url is resolved in such a way:

[https://host.com/project/spark/proxy/app-20221107152648-0484]

And I've got timeout every single time

 

However similar URLs like

[https://host.com/project/spark/proxy/worker-20221025101018-xxx|https://datahack.corpnet.pl/cvm/spark/proxy/worker-20221025101018-172.26.0.15-45263]

works fine.

 

What can I change to get application detail UI?

 

Thanks for feedback,

 

Pawel "Paul" Granat