av9ash/GitBugs huggingface dataset:
Bug: setParams() only sets explicitly named params. Is this intentional or a bug?
Status: Open | Priority: Minor | Resolution: 
Created: 16/Mar/23 13:55
Affects Version/s: 3.3.2

The Python signature/docstring of the setParams() method for the estimators and transformers under pyspark.ml imply that if you don't set any of the named params then they will be reset to their default values.

Example from [https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.clustering.GaussianMixture.html#pyspark.ml.clustering.GaussianMixture.setParams] :
{code:java}
setParams(self, \*, featuresCol="features", predictionCol="prediction", k=2, probabilityCol="probability", tol=0.01, maxIter=100, seed=None, aggregationDepth=2, weightCol=None){code}
In the extreme this would imply that if you called setParams() with no args then _all_ the params would be reset to their default values.

But what actually happens is that _only_ the params passed in the call get changed; the values of any other params aren't affected. So if you call setParams() with no args then _no_ params get changed!

So is this behavior by design? I guess it is from the name of the method. But it is counter-intuitive from its docstring. So if this behavior is intentional then perhaps the default docstring should make this explicit by saying something like:

"Sets the named params. The values of other params are not affected."
Bug: spark-submit - issue when resolving dependencies hosted on a private repository in kubernetes cluster mode
Status: Open | Priority: Minor | Resolution: 
Created: 17/Mar/23 11:19
Affects Version/s: 3.3.2

When using [spark operator|https://github.com/GoogleCloudPlatform/spark-on-k8s-operator], if dependencies are hosted on a private repository with authentication needed (like S3 or OCI) the spark operator submitting the job need to have all the secrets to access all dependencies. If not the spark-submit fails.

On a multi tenant kubernetes cluster where the spark operator and spark jobs execution are on seperate namespaces, it involves duplicating all secrets or it won't work.

It seems that spark-submit need to acces dependencies (with credentials) only to resolveGlobPath ([https://github.com/apache/spark/blob/v3.3.2/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L364-L367)] . It seems to me (but need to be confirmed by someone more skilled than me on spark internals behavior) that this resolveGlobPath task is also done when the driver is downloading the jars.

Would it be possible to have this resolveGlobPath task skipped when running on a  Kubernetes Cluster in cluster mode ?

For example add a condition like this arround the 364-367 lines :
{code:java}
if (isKubernetesCluster) {
...
} {code}
We could even, for compatibility reason with old behavior if needed, add also a condition on a spark parameter like this :
{code:java}
if (isKubernetesCluster && sparkConf.getBoolean("spark.kubernetes.resolevGlobPathsInSubmit", true)) { 
...
}{code}
i tested both solution locally and it seems to resolve the case.

Do yout think I need to consider other elements ?

I may submit a patch depending on your feedback
Bug: Use getParameterCount function instead of getParameterTypes.length
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Mar/23 07:12
Affects Version/s: 3.3.3

Since jdk1.8 there is an additional function in reflection API {{{}getParameterCount{}}}, it is better to use that function instead of {{getParameterTypes.length}} because {{getParameterTypes}} function makes a copy of the parameter types array every invocation.

This will help to avoid redundant arrays creation.
Bug: Upgrade some maven-plugins
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Mar/23 15:10
Affects Version/s: 3.5.0

maven-enforcer-plugin 3.0.0-M2 -> 3.2.1
 - [https://github.com/apache/maven-enforcer/releases/tag/enforcer-3.2.1]
 - [https://github.com/apache/maven-enforcer/releases/tag/enforcer-3.1.0]

build-helper-maven-plugin 3.2.0 -> 3.3.0
 - [https://github.com/mojohaus/build-helper-maven-plugin/releases/tag/build-helper-maven-plugin-3.3.0]

maven-compiler-plugin 3.10.1 -> 3.11.0
 - [https://github.com/apache/maven-compiler-plugin/releases/tag/maven-compiler-plugin-3.11.0]

maven-surefire-plugin 3.0.0-M9 -> 3.0.0
 - [https://github.com/apache/maven-surefire/releases/tag/surefire-3.0.0]
maven-javadoc-plugin 3.4.1 -> 3.5.0
 - [https://github.com/apache/maven-javadoc-plugin/releases/tag/maven-javadoc-plugin-3.5.0]

maven-deploy-plugin 3.0.0 -> 3.1.0
 - [https://github.com/apache/maven-deploy-plugin/releases/tag/maven-deploy-plugin-3.1.0]
Bug: Show result expressions in AggregateExec
Status: In Progress | Priority: Minor | Resolution: 
Created: 17/Mar/23 02:50
Affects Version/s: 3.4.0

If the result expressions in AggregateExec are not empty, we should display them. Or we will get confused because some important expressions do not show up in the DAG.

For example, the plan for query *SELECT sum(p) from values(cast(23.4 as decimal(7,2))) t(p)*  was incorrect because the result expression *MakeDecimal(sum(UnscaledValue(p#0))#1L,17,2) AS sum(p)#2* is not displayed

Before 

{code:java}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[], functions=[sum(UnscaledValue(p#0))], output=[sum(p)#2])
   +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=11]
      +- HashAggregate(keys=[], functions=[partial_sum(UnscaledValue(p#0))], output=[sum#5L])
         +- LocalTableScan [p#0]
{code}


After

{code:java}
== Physical Plan ==     
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[], functions=[sum(UnscaledValue(p#0))], results=[MakeDecimal(sum(UnscaledValue(p#0))#1L,17,2) AS sum(p)#2], output=[sum(p)#2])
   +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=38]
      +- HashAggregate(keys=[], functions=[partial_sum(UnscaledValue(p#0))], results=[sum#13L], output=[sum#13L])
         +- LocalTableScan [p#0]
{code}
Bug: Unprintable IllegalArgumentException with Hive catalog enabled in "Hadoop Free" distibution
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Mar/23 19:18
Affects Version/s: 3.5.0

Reproduction steps:
1. download a standard "Hadoop Free" build
2. Start pyspark REPL with Hive support
{code:java}
SPARK_DIST_CLASSPATH=$(~/dist/hadoop-3.4.0-SNAPSHOT/bin/hadoop classpath) ~/dist/spark-3.2.3-bin-without-hadoop/bin/pyspark --conf spark.sql.catalogImplementation=hive
{code}
3. Execute any simple dataframe operation
{code:java}
>>> spark.range(100).show()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/dist/spark-3.2.3-bin-without-hadoop/python/pyspark/sql/session.py", line 416, in range
    jdf = self._jsparkSession.range(0, int(start), int(step), int(numPartitions))
  File "/home/user/dist/spark-3.2.3-bin-without-hadoop/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/home/user/dist/spark-3.2.3-bin-without-hadoop/python/pyspark/sql/utils.py", line 117, in deco
    raise converted from None
pyspark.sql.utils.IllegalArgumentException: <exception str() failed>
{code}
4. In fact you can just call spark.conf to trigger this issue
{code:java}
>>> spark.conf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
...
{code}

There are probably two issues here:
1) that Hive support should be gracefully disabled if it the dependency not on the classpath as claimed by https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html
2) but at the very least the user should be able to see the exception to understand the issue, and take an action
Bug: Remove aliases from deprecated numpy data types
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Mar/23 07:02
Affects Version/s: 3.3.0

Numpy has started changing the alias to some of its data-types. This means that users with the latest version of numpy they will face either warnings or errors according to the type that they are using. This affects all the users using numoy > 1.20.0. One of the types was fixed back in September with this [pull|https://github.com/apache/spark/pull/37817] request.

The problem can be split into 2 types:

[numpy 1.24.0|https://github.com/numpy/numpy/pull/22607]: The scalar type aliases ending in a 0 bit size: np.object0, np.str0, np.bytes0, np.void0, np.int0, np.uint0 as well as np.bool8 are now deprecated and will eventually be removed. At this point in numpy 1.25.0 they give a awarning
[numpy 1.20.0|https://github.com/numpy/numpy/pull/14882]: Using the aliases of builtin types like np.int is deprecated and removed since numpy version 1.24.0

The changes are needed so pyspark can be compatible with the latest numpy and avoid
 * attribute errors on data types being deprecated from version 1.20.0: [https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations]
 * warnings on deprecated data types from version 1.24.0: [https://numpy.org/devdocs/release/1.24.0-notes.html#deprecations]

 

From my main research I see the following:

The only changes that are functional are related with the conversion.py file. The rest of the changes are inside tests in the user_guide or in some docstrings describing specific functions. Since I am not an expert in these tests I wait for the reviewer and some people with more experience in the pyspark code.

These types are aliases for classic python types so yes they should work with all the numpy versions [1|https://numpy.org/devdocs/release/1.20.0-notes.html], [2|https://stackoverflow.com/questions/74844262/how-can-i-solve-error-module-numpy-has-no-attribute-float-in-python]. The error or warning comes from the call to the numpy.

 

For the versions I chose to include from 3.3 and onwards but I see that 3.2 also is still in the 18 month maintenace cadence as it was released in October 2021.

 

The pull request: [https://github.com/apache/spark/pull/40220]

Best Regards,
Aimilios
Bug: Increase the lockAcquireTimeoutMs for acquiring the RocksDB state store in Structure Streaming
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Mar/23 22:05
Affects Version/s: 3.5.0

We are seeing query failure which is caused by RocksDB acquisition failure for the retry tasks.
 *  at t1, we shrink the cluster to only have one executor

{code:java}
23/03/05 22:47:21 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20230305224215-0000/2 is now DECOMMISSIONED (worker decommissioned because of kill request from HTTP endpoint (data migration disabled))
23/03/05 22:47:21 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20230305224215-0000/3 is now DECOMMISSIONED (worker decommissioned because of kill request from HTTP endpoint (data migration disabled))
{code}
 
 * at t1+2min, task 7 at its first attempt (i.e. task 7.0) is scheduled to the alive executor

{code:java}
23/03/05 22:49:58 INFO TaskSetManager: Starting task 7.0 in stage 133.0 (TID 685) (10.166.225.249, executor 0, partition 7, ANY, {code}
 

It seems that task 7.0 is able to pass [*{{dataRDD.iterator(partition, ctxt)}}*|https://github.com/apache/spark/blob/4db8e7b7944302a3929dd6a1197ea1385eecc46a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala#L123] and acquires the rocksdb lock as we are seeing
{code:java}
23/03/05 22:51:59 WARN TaskSetManager: Lost task 4.1 in stage 133.1 (TID 700) (10.166.225.249 executor 0): java.lang.IllegalStateException: StateStoreId(opId=0,partId=7,name=default): RocksDB instance could not be acquired by [ThreadId: Some(50), task: partition 7.1 in stage 133.1, TID 700] as it was not released by [ThreadId: Some(449), task: partition 7.0 in stage 133.0, TID 685] after 60003 ms.
23/03/05 22:52:59 WARN TaskSetManager: Lost task 4.2 in stage 133.1 (TID 702) (10.166.225.249 executor 0): java.lang.IllegalStateException: StateStoreId(opId=0,partId=7,name=default): RocksDB instance could not be acquired by [ThreadId: Some(1495), task: partition 7.2 in stage 133.1, TID 702] as it was not released by [ThreadId: Some(449), task: partition 7.0 in stage 133.0, TID 685] after 60006 ms.
23/03/05 22:53:59 WARN TaskSetManager: Lost task 4.3 in stage 133.1 (TID 704) (10.166.225.249 executor 0): java.lang.IllegalStateException: StateStoreId(opId=0,partId=7,name=default): RocksDB instance could not be acquired by [ThreadId: Some(46), task: partition 7.3 in stage 133.1, TID 704] as it was not released by [ThreadId: Some(449), task: partition 7.0 in stage 133.0, TID 685] after 60003 ms.
{code}
 
Increasing the *lockAcquireTimeoutMs* to 2 minutes such that 4 task retries will give us 8 minutes to acquire the lock and it is larger than connectionTimeout with retries (3 * 120s).
Bug: build/sbt usage error messages and shellcheck warn/error
Status: Closed | Priority: Minor | Resolution: Not A Problem
Created: 08/Mar/23 06:33
Affects Version/s: 3.3.2

The build/sbt tool's usage information has some missing content:
 
{code:java}
(base) spark% ./build/sbt -help
Usage:  [options]

  -h | -help         print this message
  -v | -verbose      this runner is chattier
{code}

And also some shellcheck warn/error.
Bug: CSV Reader - multiline without quoted fields
Status: Open | Priority: Minor | Resolution: 
Created: 03/Mar/23 11:16
Affects Version/s: 3.3.1

Hello,

We are facing an issue with the CSV format.
When we try to read a "multiline file without quoted fields" the expected result is not good.

With quoted fields, all is ok. ( cf the screenshot ) 

You can reproduce it easily with this code (just replace file path ) :
{code:java}
spark.read.options(Map(
        "multiline" -> "true",
        "quote" -> "",
        "header" -> "true",
      )).csv("/Users/fferreira/correct_multiline.csv").show(false)

spark.read.options(Map(
        "multiline" -> "true",
        "header" -> "true",      )).csv("/Users/fferreira/correct_multiline_with_quote.csv").show(false)
{code}
We continue to investigate on our side.

Thanks you.

!image-2023-03-03-12-11-21-258.png!
Bug: Display executor remove reason statistics for clusters which don't have dynamic allocation
Status: Open | Priority: Minor | Resolution: 
Created: 09/Mar/23 14:56
Affects Version/s: 3.3.2

The remove reason statistics do not seem to be enabled for clusters which do not have dynamic allocation enabled. For clusters which do have dynamic allocation, we see this:

{code:bash}
ExecutorMonitor: Executor 2 is removed. Remove reason statistics: (gracefully decommissioned: 2, decommision unfinished: 0, driver killed: 0, unexpectedly exited: 0).
{code}


Something similar would be helpful for clusters which don't have dynamic allocation, as executors can still be killed for whatever reason and it would be helpful to see if those killed executors were able to gracefully decommission.
Bug: when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config
Status: Open | Priority: Minor | Resolution: 
Created: 09/Mar/23 09:33
Affects Version/s: 3.2.1

*spark config :* 
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
 
*Sql script:* 
 
{color:#4c9aff}*create* *table* spark_catalog.test.iceberg5(id {*}bigint{*}, *data* string) *USING* iceberg;{color}

{color:#4c9aff}*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 'c');{color}

{color:#4c9aff}*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;{color}

{color:#4c9aff}*drop* *table* spark_catalog.test.iceberg5 PURGE;{color}

*spark web :*  

!image-2023-03-09-17-34-26-428.png!
Bug: Better error message for unspported `Pivot` operator in Structure Streaming
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Mar/23 18:40
Affects Version/s: 3.5.0

{{pivot}} is an unsupported operation in structured streaming but produces a bad error message that is quite misleading.

 

The following is the current error message for the {{pivot}} in SS:
{{AnalysisException: Queries with streaming sources must be executed with writeStream.start();}}
Bug: Upgrad cyclonedx from 2.7.3 to 2.7.5
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 02/Mar/23 05:15
Affects Version/s: 3.5.0

!https://user-images.githubusercontent.com/15246973/222338040-d7c8d595-be0b-40bb-af49-6b260dc0c425.png!
Bug: Avoid documenting None as as a return value in docstring
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Mar/23 04:32
Affects Version/s: 3.4.0

e.g.):

{code}
+++ b/python/pyspark/sql/dataframe.py
@@ -385,10 +385,6 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
         name : str
             Name of the view.

-        Returns
-        -------
-        None
-
         Examples
         --------
         Create a local temporary view named 'people'.
@@ -426,10 +422,6 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
         name : str
             Name of the view.

-        Returns
-        -------
-        None
-
         Examples
         --------
         Create a global temporary view.
@@ -467,10 +459,6 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
         name : str
             Name of the view.

-        Returns
-        -------
-        None
{code}

to be consistent. In Python, it's idiomatic to don't specify the return for return None.
Bug: Improve the yarn document for lo4j2 configuration
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Mar/23 07:19
Affects Version/s: 3.3.2

Since Spark3.3 has changed log4j1 to log4j2, some documents should also be updated. 

For example, docs/running-on-yarn.md still uses log4j1 syntax as follows.

`log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log`.
Bug: Minor grammatical change to "Supports Spark Connect" message
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 13/Mar/23 15:21
Affects Version/s: 3.4.0

Changing "Support Spark Connect" to "Supports Spark Connect" in the 3.4.0 version change message which is also used in the documentation:

 
.. versionchanged:: 3.4.0
     Supports Spark Connect.
Bug: The generated protobuf java file is too large
Status: In Progress | Priority: Trivial | Resolution: 
Created: 08/Mar/23 03:52
Affects Version/s: 3.5.0

Our project used generated protobuf java file too large so can't be index by IDEA. So I can't run program with IDEA. The way to fix this is change IDEA  idea.max.intellisense.filesize value to 10000. I can't find how to fix this in project README before I google. So I want to tell other new guys in Spark.
Bug: Upgrade dropwizard metrics 4.2.17
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 02/Mar/23 16:09
Affects Version/s: 3.5.0

* [https://github.com/dropwizard/metrics/releases/tag/v4.2.16]
 * [https://github.com/dropwizard/metrics/releases/tag/v4.2.17]
Bug: Spark DataFrame processed the same partition twice, which can be reproduced with very simple code.
Status: Open | Priority: Critical | Resolution: 
Created: 29/Mar/24 09:33
Affects Version/s: 3.5.1

The data in Partition 0 was executed twice.

Here is the reproduction code; the issue occurs every time.
It appears that the string starting with "0cool" is printed twice.

 
{code:java}
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("llm hard negs records") \
    .master(f"local[8]") \
    .getOrCreate()


def process_partition(index, partition):
    results = []
    s = 0

    for _ in partition:
        row = {"Result": "cool"}
        results.append(row)
        s += 1

    print(str(index) + "cool" + str(s))
    return results


data = list(range(2000))

results_rdd = spark.sparkContext.parallelize(data).repartition(8).mapPartitionsWithIndex(process_partition)
results_df = results_rdd.toDF(["Query"])

output_path = "/tmp/bc_inputs6"
results_df.write.json(output_path, mode="overwrite")
{code}
Bug: PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 06/Mar/24 12:59
Affects Version/s: 3.4.0

This seems to be a very old bug in optimizer. Related ticket:  https://issues.apache.org/jira/browse/SPARK-21765

When filter is evaluated to be always false, PruneFilters replaces the filter with empty LocalRelation, which effectively prunes filter. The logic cares about migration of the isStreaming flag, but incorrectly migrated in some case, via picking up the value of isStreaming flag from root node rather than filter (or child).

isStreaming flag is true if the value of isStreaming flag from any of children is true. Flipping the coin, some children might have isStreaming flag as "false". If the filter being pruned is a descendant to such children (in other word, ancestor of streaming node), LocalRelation is incorrectly tagged as streaming where it should be batch.
Bug: SPIP: Structured Logging Framework for Apache Spark
Status: Open | Priority: Major | Resolution: 
Created: 01/Mar/24 04:46
Affects Version/s: 4.0.0

This proposal aims to enhance Apache Spark's logging system by implementing structured logging. This transition will change the format of the default log files from plain text to JSON, making them more accessible and analyzable. The new logs will include crucial identifiers such as worker, executor, query, job, stage, and task IDs, thereby making the logs more informative and facilitating easier search and analysis.
h2. Current Logging Format

The current format of Spark logs is plain text, which can be challenging to parse and analyze efficiently. An example of the current log format is as follows:
{code:java}
23/11/29 17:53:44 ERROR BlockManagerMasterEndpoint: Fail to know the executor 289 is alive or not.
org.apache.spark.SparkException: Exception thrown in awaitResult: 
    <stacktrace…>
Caused by: org.apache.spark.rpc.RpcEndpointNotFoundException: ..
{code}
h2. Proposed Structured Logging Format

The proposed change involves structuring the logs in JSON format, which organizes the log information into easily identifiable fields. Here is how the new structured log format would look:
{code:java}
{
   "ts":"23/11/29 17:53:44",
   "level":"ERROR",
   "msg":"Fail to know the executor 289 is alive or not",
   "context":{
       "executor_id":"289"
   },
   "exception":{
      "class":"org.apache.spark.SparkException",
      "msg":"Exception thrown in awaitResult",
      "stackTrace":"..."
   },
   "source":"BlockManagerMasterEndpoint"
} {code}
This format will enable users to upload and directly query driver/executor/master/worker log files using Spark SQL for more effective problem-solving and analysis, such as tracking executor losses or identifying faulty tasks:
{code:java}
spark.read.json("hdfs://hdfs_host/logs").createOrReplaceTempView("logs")
/* To get all the executor lost logs */
SELECT * FROM logs WHERE contains(message, 'Lost executor');
/* To get all the distributed logs about executor 289 */
SELECT * FROM logs WHERE executor_id = 289;
/* To get all the errors on host 100.116.29.4 */
SELECT * FROM logs WHERE host = "100.116.29.4" and log_level="ERROR";
{code}
 

SPIP doc: [https://docs.google.com/document/d/1rATVGmFLNVLmtxSpWrEceYm7d-ocgu8ofhryVs4g3XU/edit?usp=sharing]