av9ash/GitBugs huggingface dataset:
Bug: Incorrect column resolution on FULL OUTER JOIN with USING
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/23 19:48
Affects Version/s: 3.3.2

This was tested on Spark 3.3.2 and Spark 3.4.0.

{code}
Causes [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `aws_dbr_a`.`key` cannot be resolved. Did you mean one of the following? [`key`].; line 4, pos 7
{code}


FULL OUTER JOIN with USING and/or the WHERE seems relevant since I can get the query to work with any of these modifications. 


{code}
# -- FULL OUTER JOIN
                   WITH
                   aws_dbr_a AS (select key from values ('a') t(key)),
                   gcp_pro_b AS (select key from values ('a') t(key))
                   SELECT aws_dbr_a.key
                   FROM aws_dbr_a
                   FULL OUTER JOIN gcp_pro_b USING (key)
                   WHERE aws_dbr_a.key NOT LIKE 'spark.clusterUsageTags.%';
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `aws_dbr_a`.`key` cannot be resolved. Did you mean one of the following? [`key`].; line 4 pos 7
# -- INNER JOIN
                   WITH
                   aws_dbr_a AS (select key from values ('a') t(key)),
                   gcp_pro_b AS (select key from values ('a') t(key))
                   SELECT aws_dbr_a.key
                   FROM aws_dbr_a
                   JOIN gcp_pro_b USING (key)
                   WHERE aws_dbr_a.key NOT LIKE 'spark.clusterUsageTags.%';
+-----+
| key |
|-----|
| a   |
+-----+
1 row in set
Time: 0.507s
# -- NO Filter
                   WITH
                   aws_dbr_a AS (select key from values ('a') t(key)),
                   gcp_pro_b AS (select key from values ('a') t(key))
                   SELECT aws_dbr_a.key
                   FROM aws_dbr_a
                   FULL OUTER JOIN gcp_pro_b USING (key);
+-----+
| key |
|-----|
| a   |
+-----+
1 row in set
Time: 1.021s
# -- ON instead of USING
                   WITH
                   aws_dbr_a AS (select key from values ('a') t(key)),
                   gcp_pro_b AS (select key from values ('a') t(key))
                   SELECT aws_dbr_a.key
                   FROM aws_dbr_a
                   FULL OUTER JOIN gcp_pro_b ON aws_dbr_a.key = gcp_pro_b.key
                   WHERE aws_dbr_a.key NOT LIKE 'spark.clusterUsageTags.%';
+-----+
| key |
|-----|
| a   |
+-----+
1 row in set
Time: 0.514s
{code}
Bug: Support to set  Python executable in workers during runtime
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/23 10:20
Affects Version/s: 3.5.0

For Python UDF, we can support runtime configuration to set Python workers. It will create a new Python worker if the configuration is changed.

This is especially useful when you want to run Spark with different dependencies during runtime. See also https://www.databricks.com/blog/2020/12/22/how-to-manage-python-dependencies-in-pyspark.html
Bug: Update "Supported Pandas API" page to point out the proper pandas docs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/23 01:36
Affects Version/s: 3.4.0

[https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/supported_pandas_api.html#supported-pandas-api] not point out the wrong pandas version.
Bug: bitmap function
Status: Open | Priority: Major | Resolution: 
Created: 12/May/23 16:29
Affects Version/s: 3.4.0

maybe spark need to have some bitmap functions？ example  like bitmapBuild 、bitmapAnd、bitmapAndCardinality in clickhouse or other OLAP engine。

This is often used in user profiling applications but i don't find in spark

 

 
h2.
Bug: Skip compiling useless files when making distribution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/23 14:19
Affects Version/s: 3.5.0

-Dmaven.javadoc.skip=true to skip java doc
-Dskip=true to skip scala doc. Please see: https://davidb.github.io/scala-maven-plugin/doc-jar-mojo.html#skip
-Dmaven.source.skip to skip build sources.jar
-Dmaven.test.skip to skip build test-jar
-Dcyclonedx.skip=true to skip making bom. Please see: https://cyclonedx.github.io/cyclonedx-maven-plugin/makeBom-mojo.html#skip
Bug: Memory leak in Spark UI
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/May/23 09:36
Affects Version/s: 2.4.4

We have a distributed Spark application running on Azure HDInsight using Spark version 2.4.4.

After a few days of active processing on our application, we have noticed that the GC CPU time ratio of the driver is close to 100%. We suspected a memory leak. Thus, we have produced a heap dump and analyzed it using Eclipse Memory Analyzer.

Here is some interesting data from the driver's heap dump (heap size is 8 GB):
 * The estimated retained heap size of String objects (~5M instances) is 3.3 GB. It seems that most of these instances correspond to spark events.
 * Spark UI's AppStatusListener instance estimated retained size is 1.1 GB.
 * The number of LiveJob objects with status "RUNNING" is 18K, knowing that there shouldn't be more than 16 live running jobs since we use a fixed size thread pool of 16 threads to run spark queries.
 * The number of LiveTask objects is 485K.
 * The AsyncEventQueue instance associated to the AppStatusListener has a value of 854 for dropped events count and a value of 10001 for total events count, knowing that the dropped events counter is reset every minute and that the queue's default capacity is 10000.

We think that there is a memory leak in Spark UI. Here is our analysis of the root cause of this leak:
 * AppStatusListener is notified of Spark events using a bounded queue in AsyncEventQueue.
 * AppStatusListener updates its state (kvstore, liveTasks, liveStages, liveJobs, ...) based on the received events. For example, onTaskStart adds a task to liveTasks map and onTaskEnd removes the task from liveTasks map.
 * When the rate of events is very high, the bounded queue in AsyncEventQueue is full, some events are dropped and don't make it to AppStatusListener.
 * Dropped events that signal the end of a processing unit prevent the state of AppStatusListener from being cleaned. For example, a dropped onTaskEnd event, will prevent the task from being removed from liveTasks map, and the task will remain in the heap until the driver's JVM is stopped.

We were able to confirm our analysis by reducing the capacity of the AsyncEventQueue (spark.scheduler.listenerbus.eventqueue.capacity=10). After having launched many spark queries using this config, we observed that the number of active jobs in Spark UI increased rapidly and remained high even though all submitted queries have completed. We have also noticed that some executor task counters in Spark UI were negative, which confirms that AppStatusListener state does not accurately reflect the reality and that it can be a victim of event drops.

Suggested fix:
There are some limits today on the number of "dead" objects in AppStatusListener's maps (for example: spark.ui.retainedJobs). We suggest enforcing another configurable limit on the number of total objects in AppStatusListener's maps and kvstore. This should limit the leak in the case of high events rate, but AppStatusListener stats will remain inaccurate.
Bug: Expand QueryTerminatedEvent to contain error class if it exists in exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/23 01:45
Affects Version/s: 3.5.0

As of now, the event QueryTerminatedEvent has a field about exception which is just a string representation of underlying exception, mostly stack trace. That said, people relying on streaming query listener can't get any benefit from the movement for employing error class framework. This is a bad observability once we migrate exceptions in streaming query to error class framework where exceptions are well categorized.

We can expand the event of QueryTerminatedEvent - if the exception contains error class information, we can ship this along with existing string version of exception (message).
Bug: IN subquery ListQuery has wrong nullability
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/23 21:12
Affects Version/s: 3.4.0

IN subquery expressions are incorrectly marked as non-nullable, even when they are actually nullable. They correctly check the nullability of the left-hand-side, but the right-hand-side of a IN subquery, the ListQuery, is currently defined with nullability = false always. This is incorrect and can lead to incorrect query transformations.

Example: (non_nullable_col IN (select nullable_col)) <=> TRUE . Here the IN expression returns NULL when the nullable_col is null, but our code marks it as non-nullable, and therefore SimplifyBinaryComparison transforms away the <=> TRUE, transforming the expression to non_nullable_col IN (select nullable_col) , which is an incorrect transformation because NULL values of nullable_col now cause the expression to yield NULL instead of FALSE.

This bug can potentially lead to wrong results, but in most cases this doesn't directly cause wrong results end-to-end, because IN subqueries are almost always transformed to semi/anti/existence joins in RewritePredicateSubquery, and this rewrite can also incorrectly discard NULLs, which is another bug. But we can observe it causing wrong behavior in unit tests, and it could easily lead to incorrect query results if there are changes to the surrounding context, so it should be fixed regardless.

This is a long-standing bug that has existed at least since 2016, as long as the ListQuery class has existed.
Bug: Confused errors from the DATEADD function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/23 06:42
Affects Version/s: 3.4.0

The code example portraits the issue:

{code:sql}
spark-sql (default)> select dateadd('MONTH', 1, date'2023-05-11');
[WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `dateadd` requires 2 parameters but the actual number is 3. Please, refer to 'https://spark.apache.org/docs/latest/sql-ref-functions.html' for a fix.; line 1 pos 7
{code}

The error says about number of arguments passed to DATEADD but the issue is about the type of the first argument.
Bug: The status of free pages in allocatedPages mismatch with pageTable after invoking TaskMemoryManager::cleanUpAllAllocatedMemory
Status: Open | Priority: Major | Resolution: 
Created: 15/May/23 08:38
Affects Version/s: 3.4.0

For TaskMemoryManager,  `allocatePags`  is a Bitmap for tracking free pages in `pageTables`.

But inner TaskMemoryManager::cleanUpAllAllocatedMemory, it fills up `pageTables` with null but not clear up the `allocatePags`. It leads to the status of free pages mismatch between them.
Bug: spark on k8s supports spark.executor.extraLibraryPath 
Status: Open | Priority: Major | Resolution: 
Created: 15/May/23 08:19
Affects Version/s: 3.2.3

For extraLibraryPath to work as expected, the executor JVM should be started with prefixed  env(assumed on Linux) such as:

LD_LIBRARY_PATH=LD_LIBRARY_PATH:${spark.executor.extraLibraryPath} bin/java xxxx

 

However there are two problems for running on K8S:
 # spark on k8s doesn't handle `spark.executor.extraLibraryPath` at all
 # k8s currently don't have a way to expand environment variable correctly
Bug: SparkConf's get and getAllWithPrefix support variable substitution
Status: Open | Priority: Major | Resolution: 
Created: 11/May/23 09:21
Affects Version/s: 3.2.4

After SPARK-16272:

variable substitution is available for `ConfigEntry`s, this creates some confusion for get conf directly vs get conf from ConfigEntry. For example:
{code:java}
// suppose spark.a=${env:var_a} is set in sparkConf, and it's defined as a ConfigEntry as CONFIG_ENTRY_A

// this would return ${env:var_a}
val a = sparkConf.get("spark.a") 

// this would return the value of var_a in the current JVM process
val configA = sparkConf.get(CONFIG_ENTRY_A)

{code}
I'd like to propose that we should support variable substitution for get config directly.

 

I noticed this problem when running spark applications on k8s, and noticed some substitutions are not work as expected. For k8s particularly, it uses getAllWithPrefix extensively, which also suffers this problem
Bug: Define the DATE_ADD and DATE_DIFF functions with 3-args
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/23 11:12
Affects Version/s: 3.4.0

Spark supports the DATE_ADD and DATE_DIFF functions with 2 arguments but when an user calls the same functions with 3 arguments, Spark SQL outputs the confusing error:

{code:sql}
spark-sql (default)> select date_add(MONTH, 1, date'2023-05-13');
[UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column or function parameter with name `MONTH` cannot be resolved. ; line 1 pos 16;
'Project [unresolvedalias('date_add('MONTH, 1, 2023-05-13), None)]
+- OneRowRelation
{code}
Bug: Improve list of suggested column/attributes in `UNRESOLVED_COLUMN.WITH_SUGGESTION` error class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/23 17:13
Affects Version/s: 3.4.0

Match the style of unresolved column/attribute when sorting list of suggested columns. If an unresolved column name is single-part identifier - use same style for suggested columns.
Bug: Mexico has changed observation of DST, this breaks timestamp_utc()
Status: Open | Priority: Major | Resolution: 
Created: 11/May/23 21:52
Affects Version/s: 3.4.0

[https://www.timeanddate.com/time/change/mexico/mexico-city?year=2023#:~:text=Daylight%20Saving%20Time%20(DST)%20Not,was%20on%20October%2030%2C%202022.]

Mexico has stopped observing DST. This results in wrong results for:
from_utc_timestamp([timestamp], 'America/Mexico_City')
Bug: Allow deserializing protobuf enum fields as integers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/23 21:35
Affects Version/s: 3.4.0

When deserializing protobuf enum fields, the spark-protobuf library will deserialize them as string values based on the enum name in the proto. E.g. 
{code:java}
message Person {
  enum Job {
    NOTHING = 0;
    ENGINEER = 1;
    DOCTOR = 2;
  }
  Job job = 1;
}{code}
And we have a message like
{code:java}
Person(job=ENGINEER){code}
Then the deserialized value will be:
{code:java}
{"job": "ENGINEER"}{code}
However it can be useful to deserialize the enum integer value rather than the name (and this option exists in other major libraries). So, namely:
{code:java}
{"job": 1}{code}
 

Examples in other libraries:
 * protobuf-java-util JsonFormat: [https://javadoc.io/doc/com.google.protobuf/protobuf-java-util/3.10.0/com/google/protobuf/util/JsonFormat.Printer.html#printingEnumsAsInts--]
 * golang/protobuf jsonpb marshaler [https://pkg.go.dev/github.com/golang/protobuf/jsonpb#Marshaler]

I propose extending spark-protobuf to add this functionality.
Bug: [K8S] Make limit.cores be able to be fallen back to request.cores
Status: Open | Priority: Major | Resolution: 
Created: 09/May/23 04:12
Affects Version/s: 3.4.0

make limit.cores be able to be fallen back to request.cores

now without limit.cores, we will meet below issue:
{code:java}
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://****/api/v1/namespaces/hadooptessns/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "****" is forbidden: failed quota: high-qos-limit-requests: must specify limits.cpu. {code}
If spark.kubernetes.executor/driver.limit.cores is not specified, how about treating request.cores as limit.cores?
Bug: Fix flakiness in Kafka RDD suites due to port binding configuration issue
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/23 21:21
Affects Version/s: 3.4.0

In SPARK-36837 we updated Kafka to 3.10, which uses a different set of configuration options for configuring the broker listener port. That PR only updated one of two KafkaTestUtils files (the SQL one), so the other one (used by Core tests) had an ineffective port binding configuration and would bind to the default 9092 port. This could lead to flakiness if multiple suites binding to that port ran in parallel.

To fix this, we just need to copy the updated port binding configuration from the other suite.
Bug: Add documentation for protobuf deserialization options
Status: Open | Priority: Major | Resolution: 
Created: 08/May/23 12:55
Affects Version/s: 3.4.0

Follow-up from PR comment: [https://github.com/apache/spark/pull/41075#discussion_r1186958551]

 

Currently theres no documentation here [https://github.com/apache/spark/blob/master/docs/sql-data-sources-protobuf.md#deploying] for the full range of options available for `from_protobuf`, so users would need to discover these options by reading the scala code directly. Let's update the documentation to include information about these options.
Bug: Add config to control threshold of unregister map output when fetch failed
Status: Open | Priority: Major | Resolution: 
Created: 07/May/23 19:24
Affects Version/s: 3.4.0

Spark will unregister map output on the executor when fetch failed from this executor. This might be too aggressive when fetch failed is temporary and recoverable. So adding the config to control the number of fetch failed failures needed to unregister map output and allow to disable unregister.
Bug: Sequence expression can overflow
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 06/May/23 03:04
Affects Version/s: 3.5.0

Spark has a (long-standing) overflow bug in the {{sequence}} expression.

 

Consider the following operations:

{{spark.sql("CREATE TABLE foo (l LONG);")}}
{{spark.sql(s"INSERT INTO foo VALUES (${Long.MaxValue});")}}
{{spark.sql("SELECT sequence(0, l) FROM foo;").collect()}}

 

The result of these operations will be:

{{Array[org.apache.spark.sql.Row] = Array([WrappedArray()])}}

an unintended consequence of overflow.

 

The sequence is applied to values {{0}} and {{Long.MaxValue}} with a step size of {{1}} which uses a length computation defined [here|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3451]. In this calculation, with {{{}start = 0{}}}, {{{}stop = Long.MaxValue{}}}, and {{{}step = 1{}}}, the calculated {{len}} overflows to {{{}Long.MinValue{}}}. The computation, in binary looks like:

{{  0111111111111111111111111111111111111111111111111111111111111111}}

{{- 0000000000000000000000000000000000000000000000000000000000000000 }}

{{------------------------------------------------------------------      0111111111111111111111111111111111111111111111111111111111111111}}

{{/ 0000000000000000000000000000000000000000000000000000000000000001}}

{{------------------------------------------------------------------      0111111111111111111111111111111111111111111111111111111111111111}}

{{+ 0000000000000000000000000000000000000000000000000000000000000001}}

{{------------------------------------------------------------------      1000000000000000000000000000000000000000000000000000000000000000}}

{{}}

The following [check|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3454] passes as the negative {{Long.MinValue}} is still {{{}<= MAX_ROUNDED_ARRAY_LENGTH{}}}. The following cast to {{toInt}} uses this representation and [truncates the upper bits|https://github.com/apache/spark/blob/16411188c7ba6cb19c46a2bd512b2485a4c03e2c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3457] resulting in an empty length of 0.{{{}{}}}

Other overflows are similarly problematic.{{{}{}}}
Bug: driver and executors shared same Kubernetes PVC in Spark 3.4+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/23 03:25
Affects Version/s: 3.4.0

I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below error.  I would like all the executors and driver to mount the same PVC. Is this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because otherwise each of the executors will use an unique and separate PVC. 
 
Error message is "should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple executors"
 
below is how I enabled it pvc in spark 3.3 and it works, but does not work in Spark 3.4

{code:sh}
spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc 
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir 
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc 
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir 
 
{code}
Bug: Refactor Dockerfile and workflow based on base image
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/23 02:46
Affects Version/s: 3.5.0

https://github.com/docker-library/official-images/pull/13089?notification_referrer_id=NT_kwDOABp-orI0MzIwMzMwNzY5OjE3MzYzNTQ#issuecomment-1533540388

Would it be useful to save space by sharing layers by having one image from another? 🤔 Something like the *java11-ubuntu as the "base" with r and python variants FROM that and the r-python being FROM, probably, the larger one of those?

Rough example Dockerfiles

{code:java}
FROM eclipse-temurin:11-jre-focal
# user stuff, install common deps, etc
...
# download/extract spark (maybe keeping python and R files too? they seem relatively small compared to the rest)
{code}


{code:java}
# other images in separate Dockerfiles
FROM spark:3.3.0-scala2.12-java11-ubuntu
# get "/opt/spark/{python,R}/" contents if not kept in base
# install python or R (and things like R_HOME)
{code}
Bug: Support Python 3.8 in PyPy3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/23 23:52
Affects Version/s: 3.5.0

{code:java}

pypy3 version is: Python 3.8.16 (a9dbdca6fc3286b0addd2240f11d97d8e8de187a, Dec 29 2022, 11:45:13)
[PyPy 7.3.11 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
Starting test(pypy3): pyspark.sql.tests.pandas.test_pandas_cogrouped_map (temp output: /__w/spark/spark/python/target/f1cacde7-d369-48cf-a8ea-724c42872020/pypy3__pyspark.sql.tests.pandas.test_pandas_cogrouped_map__rxih6dqu.log)
Traceback (most recent call last):
  File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/__w/spark/spark/python/pyspark/__init__.py", line 59, in <module>
    from pyspark.rdd import RDD, RDDBarrier
  File "/__w/spark/spark/python/pyspark/rdd.py", line 54, in <module>
    from pyspark.java_gateway import local_connect_and_auth
  File "/__w/spark/spark/python/pyspark/java_gateway.py", line 32, in <module>
    from pyspark.serializers import read_int, write_with_length, UTF8Deserializer
  File "/__w/spark/spark/python/pyspark/serializers.py", line 69, in <module>
    from pyspark import cloudpickle
  File "/__w/spark/spark/python/pyspark/cloudpickle/__init__.py", line 1, in <module>
    from pyspark.cloudpickle.cloudpickle import *  # noqa
  File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle.py", line 56, in <module>
    from .compat import pickle
  File "/__w/spark/spark/python/pyspark/cloudpickle/compat.py", line 13, in <module>
    from _pickle import Pickler  # noqa: F401
ModuleNotFoundError: No module named '_pickle' {code}