av9ash/GitBugs huggingface dataset:
Bug: Make PySpark working with Python 3.7
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Feb/23 03:04
Affects Version/s: 3.4.0

{code}
+ ./python/run-tests --python-executables=python3
Running PySpark tests. Output is in /home/ec2-user/spark/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python modules: ['pyspark-connect', 'pyspark-core', 'pyspark-errors', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-slow', 'pyspark-resource', 'pyspark-sql', 'pyspark-streaming']
python3 python_implementation is CPython
python3 version is: Python 3.7.16
Starting test(python3): pyspark.ml.tests.test_feature (temp output: /home/ec2-user/spark/python/target/8ca9ab1a-05cc-4845-bf89-30d9001510bc/python3__pyspark.ml.tests.test_feature__kg6sseie.log)
Starting test(python3): pyspark.ml.tests.test_base (temp output: /home/ec2-user/spark/python/target/f2264f3b-6b26-4e61-9452-8d6ddd7eb002/python3__pyspark.ml.tests.test_base__0902zf9_.log)
Starting test(python3): pyspark.ml.tests.test_algorithms (temp output: /home/ec2-user/spark/python/target/d1dc4e07-e58c-4c03-abe5-09d8fab22e6a/python3__pyspark.ml.tests.test_algorithms__lh3wb2u8.log)
Starting test(python3): pyspark.ml.tests.test_evaluation (temp output: /home/ec2-user/spark/python/target/3f42dc79-c945-4cf2-a1eb-83e72b40a9ee/python3__pyspark.ml.tests.test_evaluation__89idc7fa.log)
Finished test(python3): pyspark.ml.tests.test_base (16s)
Starting test(python3): pyspark.ml.tests.test_functions (temp output: /home/ec2-user/spark/python/target/5a3b90f0-216b-4edd-9d15-6619d3e03300/python3__pyspark.ml.tests.test_functions__g5u1290s.log)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ec2-user/spark/python/pyspark/ml/tests/test_functions.py", line 21, in <module>
    from pyspark.ml.functions import predict_batch_udf
  File "/home/ec2-user/spark/python/pyspark/ml/functions.py", line 38, in <module>
    from typing import Any, Callable, Iterator, List, Mapping, Protocol, TYPE_CHECKING, Tuple, Union
ImportError: cannot import name 'Protocol' from 'typing' (/usr/lib64/python3.7/typing.py)
Had test failures in pyspark.ml.tests.test_functions with python3; see logs.
{code}
Bug: DataFrame.drop should handle multi columns properly
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 15/Feb/23 02:46
Affects Version/s: 3.4.0

{code:java}
from pyspark.sql import Row
df1 = spark.createDataFrame([(14, "Tom"), (23, "Alice"), (16, "Bob")], ["age", "name"])
df2 = spark.createDataFrame([Row(height=80, name="Tom"), Row(height=85, name="Bob")])
df1.join(df2, df1.name == df2.name, 'inner').drop('name', 'age').show()
{code}

This works in 3.3

{code:java}
+------+
|height|
+------+
|    85|
|    80|
+------+
{code}

but fails in 3.4


{code:java}
---------------------------------------------------------------------------
AnalysisException                         Traceback (most recent call last)
Cell In[1], line 4
      2 df1 = spark.createDataFrame([(14, "Tom"), (23, "Alice"), (16, "Bob")], ["age", "name"])
      3 df2 = spark.createDataFrame([Row(height=80, name="Tom"), Row(height=85, name="Bob")])
----> 4 df1.join(df2, df1.name == df2.name, 'inner').drop('name', 'age').show()

File ~/Dev/spark/python/pyspark/sql/dataframe.py:4913, in DataFrame.drop(self, *cols)
   4911     jcols = [_to_java_column(c) for c in cols]
   4912     first_column, *remaining_columns = jcols
-> 4913     jdf = self._jdf.drop(first_column, self._jseq(remaining_columns))
   4915 return DataFrame(jdf, self.sparkSession)

File ~/Dev/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
   1316 command = proto.CALL_COMMAND_NAME +\
   1317     self.command_header +\
   1318     args_command +\
   1319     proto.END_COMMAND_PART
   1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
   1323     answer, self.gateway_client, self.target_id, self.name)
   1325 for temp_arg in temp_args:
   1326     if hasattr(temp_arg, "_detach"):

File ~/Dev/spark/python/pyspark/errors/exceptions/captured.py:159, in capture_sql_exception.<locals>.deco(*a, **kw)
    155 converted = convert_exception(e.java_exception)
    156 if not isinstance(converted, UnknownException):
    157     # Hide where the exception came from that shows a non-Pythonic
    158     # JVM exception message.
--> 159     raise converted from None
    160 else:
    161     raise

AnalysisException: [AMBIGUOUS_REFERENCE] Reference `name` is ambiguous, could be: [`name`, `name`].

{code}
Bug: CONV() silently overflows returning wrong results
Status: In Progress | Priority: Critical | Resolution: 
Created: 10/Feb/23 19:03
Affects Version/s: 3.4.0

spark-sql> SELECT CONV(SUBSTRING('0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff', 3), 16, 10);

18446744073709551615

Time taken: 2.114 seconds, Fetched 1 row(s)

spark-sql> set spark.sql.ansi.enabled = true;

spark.sql.ansi.enabled true

Time taken: 0.068 seconds, Fetched 1 row(s)

spark-sql> SELECT CONV(SUBSTRING('0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff', 3), 16, 10);

18446744073709551615

Time taken: 0.05 seconds, Fetched 1 row(s)


In ANSI mode we should raise an error for sure.
In non ANSI either an error or a NULL maybe be acceptable.

Alternatively, of course, we could consider if we can support arbitrary domains since the result is a STRING again.
Bug: table directory will lost on hdfs when `INSERT OVERWRITE` faild
Status: Closed | Priority: Critical | Resolution: Cannot Reproduce
Created: 24/Feb/23 05:44
Affects Version/s: 3.2.3

{color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during execution, the table's directory will be deleted.  this is not happen in spark 3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 6000044164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow exception*{color}

{color:#4c9aff}and then :{color}

*select* * *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}

{color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to check:{color}

!image-2023-02-24-15-23-32-977.png!
Bug: SPIP: Shutting down spark structured streaming when the streaming process completed current process
Status: Open | Priority: Major | Resolution: 
Created: 18/Feb/23 10:17
Affects Version/s: 3.2.2

Spark Structured Streaming is a very useful tool in dealing with Event Driven Architecture. In an Event Driven Architecture, there is generally a main loop that listens for events and then triggers a call-back function when one of those events is detected. In a streaming application the application waits to receive the source messages in a set interval or whenever they happen and reacts accordingly.

There are occasions that you may want to stop the Spark program gracefully. Gracefully meaning that Spark application handles the last streaming message completely and terminates the application. This is different from invoking interrupts such as CTRL-C.

Of course one can terminate the process based on the following
 # query.awaitTermination() # Waits for the termination of this query, with stop() or with error

 # query.awaitTermination(timeoutMs) # Returns true if this query is terminated within the timeout in milliseconds.

So the first one above waits until an interrupt signal is received. The second one will count the timeout and will exit when timeout in milliseconds is reached.

The issue is that one needs to predict how long the streaming job needs to run. Clearly any interrupt at the terminal or OS level (kill process), may end up the processing terminated without a proper completion of the streaming process.

I have devised a method that allows one to terminate the spark application internally after processing the last received message. Within say 2 seconds of the confirmation of shutdown, the process will invoke a graceful shutdown.

{color:#000000}This new feature proposes a solution to handle the topic doing work for the message being processed gracefully, wait for it to complete and shutdown the streaming process for a given topic without loss of data or orphaned transactions{color}
Bug: Dateset operations should not resolve the analyzed logical plan again
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/23 05:36
Affects Version/s: 3.4.0

For the following query

 
{code:java}
      sql(
        """
          |CREATE TABLE app_open (
          |  uid STRING,
          |  st TIMESTAMP,
          |  ds INT
          |) USING parquet PARTITIONED BY (ds);
          |""".stripMargin)

      sql(
        """
          |create or replace temporary view group_by_error as WITH new_app_open AS (
          |  SELECT
          |    ao.*
          |  FROM
          |    app_open ao
          |)
          |SELECT
          |    uid,
          |    20230208 AS ds
          |  FROM
          |    new_app_open
          |  GROUP BY
          |    1,
          |    2
          |""".stripMargin)

      sql(
        """
          |select
          |  `uid`
          |from
          |  group_by_error
          |""".stripMargin).show(){code}
Spark will throw the following error

 

 
{code:java}
[GROUP_BY_POS_OUT_OF_RANGE] GROUP BY position 20230208 is not in select list (valid range is [1, 2]).; line 9 pos 4 {code}
 

 

This is because the logical plan is not set as analyzed and it is analyzed again. The analyzer rules about aggregation/sort ordinals are not idempotent.
Bug: Add new function filter_value
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Feb/23 13:45
Affects Version/s: 3.3.2

Doing data validation in Spark can lead to a lot of extra evaluations of expressions. This is because conditionally evaluated expressions aren't candidates for subexpression elimination. For example a simple expression such as 

{{when(validate(col), col)}}

to only keep col if it matches some condition, will lead to col being evaluated twice. And if call itself is made up of a series of expensive expressions itself, like regular expression checks, this can lead to a lot of wasted computation time.

The initial attempt to resolve this was https://issues.apache.org/jira/browse/SPARK-35564, adding support for subexpression elimination to conditionally evaluated expressions. However I have not been able to get that merged, so this is an alternative (though I believe that is still useful on top of this).

We can add a new higher order function "filter_value" that takes the column you want to validate as an argument, and then a function that runs a lambda expression returning a boolean on whether to keep that column or not. It would have the same semantics as the above when expression, except it would guarantee to only evaluate the initial column once.

An alternative would be to implement a real definition for the NullIf expression, but that would only support exact equals checks and not any generic condition.
Bug: Avoid unnecessary parquet footer reads when no filters in vectorized reader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/23 07:20
Affects Version/s: 3.4.0

Parquet footer is now read twice even if there are no filters requiring pushdown in vectorized parquet reader.
When the NameNode is under high pressure, it will cost time to read twice. Actually we can avoid this unnecessary parquet footer reads and use footer metadata in {{{}VectorizedParquetRecordReader{}}}.
Bug: Fix constructor for java.nio.DirectByteBuffer for Java 21+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/23 08:10
Affects Version/s: 3.5.0

In the latest JDK, the constructor {{DirectByteBuffer(long, int)}} was replaced with {{{}DirectByteBuffer(long, long){}}}. We just want to support both by probing for the legacy one first and falling back to the newer one second.

This change is completely transparent for the end-user, and makes sure Spark works transparently on the latest JDK as well.
Bug: Introduce conf spark.sql.parquet.inferTimestampNTZ.enabled for TimestampNTZ inference on Parquet
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 21:35
Affects Version/s: 3.4.0

Introduce conf spark.sql.parquet.inferTimestampNTZ.enabled for TimestampNTZ inference on Parquet, instead of using spark.sql.parquet.timestampNTZ.enabled which makes it impossible for TimestampNTZ writing when the flag is disabled.
Bug: Better support for Istio service mesh while running Spark on Kubernetes
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/23 01:29
Affects Version/s: 3.2.3

h3. Support for Strict MTLS

In strict MTLS Peer Authentication Istio requires each pod to be associated with a service identity (as this allows listeners to use the correct cert and chain). Without the service identity communication goes through passthrough cluster which is not permitted in strict mode. Community is still investigating communication through IPs with strict MTLS [https://github.com/istio/istio/issues/37431#issuecomment-1412831780]. Today Spark backend creates a service record for driver however executor pods register with driver using their Pod IPs. In this model therefore, TLS handshake would fail between driver and executor and also between executors. As part of this Jira we want to similarly add service records for the executor pods as well. This can be achieved by adding a ExecutorServiceFeatureStep similar to existing DriverServiceFeatureStep
h3. Allowing binding to all IPs

Before Istio 1.10 the istio-proxy sidecar was forwarding outside traffic to localhost of the pod. Thus if the application container is binding only to Pod IP the traffic would not be forwarded to it. This was addressed in 1.10 [https://istio.io/latest/blog/2021/upcoming-networking-changes]. However the old behavior is still accessible through disabling the feature flag PILOT_ENABLE_INBOUND_PASSTHROUGH. Request to remove it has had some push back [https://github.com/istio/istio/issues/37642]. In current implementation Spark K8s backend does not allow to pass bind address for driver [https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala#L35] however as part of this Jira we want to allow passing of bind address even in Kubernetes mode so long as the bind address is 0.0.0.0. This lets user choose the behavior depending on the state of PILOT_ENABLE_INBOUND_PASSTHROUGH in her Istio cluster.
h3. Better support for istio-proxy sidecar lifecycle management

In istio-enabled cluster istio-proxy sidecars would be auto-injected to driver/executor pods. If the application is ephemeral then driver and executor containers would exit, however istio-proxy container would continue to run. This causes driver/executor pods to enter NotReady state. As part of this jira we want ability to run a post stop cleanup after driver/executor container is completed. Similarly we also want to add support for a pre start up script, which can ensure for example that istio-sidecar is up before executor/driver container gets started.
Bug: Support row skipping when reading CSV files
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Feb/23 13:36
Affects Version/s: 3.3.1

Spark currently can't read CSV files that contain lines with comments or annotations above the header and data. Work-arounds include pre-processing CSVs, or using RDDs and something like {{zipWithIndex}}. But all of these increase friction for less technical users.

This issue proposes a {{skipLines}} option for Spark's CSV parser to drop a number of unwanted lines at the top of a CSV file.
Bug: spark.kubernetes.file.upload.path not deleting files under HDFS after job completes
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Feb/23 19:02
Affects Version/s: 3.2.0

In cluster mode after uploading files to HDFS location using spark.kubernetes.file.upload.path property files are not getting cleared . 

File is successfully uploaded to hdfs location in this format spark-upload-[randomUUID] using {{KubernetesUtils}} is requested to  uploadFileUri . [https://github.com/apache/spark/blob/76a134ade60a9f354aca01eaca0b2e2477c6bd43/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L310]

following is driver log  , driver is completed successfully and shutdownhook is not cleared the hdfs files.
{code:java}
23/02/16 18:06:56 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
23/02/16 18:06:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
23/02/16 18:06:56 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
23/02/16 18:06:57 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/02/16 18:06:57 INFO MemoryStore: MemoryStore cleared
23/02/16 18:06:57 INFO BlockManager: BlockManager stopped
23/02/16 18:06:57 INFO BlockManagerMaster: BlockManagerMaster stopped
23/02/16 18:06:57 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/02/16 18:06:57 INFO SparkContext: Successfully stopped SparkContext
23/02/16 18:06:57 INFO ShutdownHookManager: Shutdown hook called
23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-efb8f725-4ead-4729-a8e0-f478280121b7
23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory /spark-local2/spark-66dbf7e6-fe7e-4655-8724-69d76d93fc1f
23/02/16 18:06:57 INFO ShutdownHookManager: Deleting directory /spark-local1/spark-53aefaee-58a5-4fce-b5b0-5e29f42e337f{code}
Bug: Improve the performance of drop partitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/23 18:10
Affects Version/s: 3.3.2

Currently to drop the matching partitions, Spark will first get all matching Partition objects from Hive metastore, and just use the partition values of these Partition objects.

We can get the matching partition names instead of the partition objects for the following reasons:
1. we can also get partition values through a partition name (like a=1/b=2)
2. the byte size of partition name is much smaller than partition object, which will help improve the performance of drop partitions.
Bug: Deprecate & remove the APIs that will be removed in pandas 2.0.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/23 05:01
Affects Version/s: 3.4.0

pandas is preparing to release 2.0 which includes bunch of API changes. ([https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v2.0.0.html#removal-of-prior-version-deprecations-changes])

We also should deprecate the APIs so that we can remove the API in next release.
Bug: High level design doc for Distributed ML <> spark connect
Status: Resolved | Priority: Major | Resolution: Done
Created: 20/Feb/23 13:03
Affects Version/s: 3.4.0

Design doc:

https://docs.google.com/document/d/1LHzwCjm2SluHkta_08cM3jxFSgfF-niaCZbtIThG-H8/edit?usp=sharing
Bug: Add Destructive Iterator for SparkResult
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/23 23:58
Affects Version/s: 3.4.0

Add a destructive iterator to SparkResult. Instead of keeping everything in memory for the life time of SparkResult object, clean it up as soon as we know we are done with it. We can use this for Dataset.toLocalIterator.
Bug: Get ParseException when run sql: "SELECT 1 UNION SELECT 1;"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/23 08:46
Affects Version/s: 3.2.3

When I run sql
{code:java}
scala> spark.sql("SELECT 1 UNION SELECT 1;") {code}
I get ParseException:
{code:java}
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'SELECT' expecting {<EOF>, ';'}(line 1, pos 15)== SQL ==
SELECT 1 UNION SELECT 1;
---------------^^^  at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:266)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:127)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:51)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:77)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
  ... 47 elided
 {code}
If I run with parentheses , it works well 
{code:java}
scala> spark.sql("(SELECT 1) UNION (SELECT 1);") 
res4: org.apache.spark.sql.DataFrame = [1: int]{code}
This should be a bug
Bug: Document SS guide doc for supporting multiple stateful operators (especially chained aggregations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/23 04:53
Affects Version/s: 3.5.0

We made a change on the guide doc for SPARK-40925 via SPARK-42105, but from SPARK-42105 we only removed the section of "limitation of global watermark". That said, we haven't provided any example of new functionality, especially that users need to know about the change of SQL function (window) in chained time window aggregations.

In this ticket, we will add the example of chained time window aggregations, with introducing new functionality of SQL function.
Bug: add spark.yarn.archive/jars similar option for spark on K8S
Status: Open | Priority: Major | Resolution: 
Created: 01/Feb/23 12:06
Affects Version/s: 3.2.2

For spark on yarn,  there are `spark.yarn.archive` and `spark.yarn.jars` to distribute spark runtime jars before driver/executor starts up.

 

I'd like to propose similar functionality for spark on K8S. The benefits are:
 # accelerating workloads migration from yarn to K8S which use the above feature
 # explore new version of spark more easily without to rebuild the spark image
 # currently, there's really no other way to add additional/extension jars to executors on k8s before startup .
Bug: Support for Pandas/Arrow Functions API
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 10/Feb/23 07:55
Affects Version/s: 3.4.0

There are derivative APIs which depend on the implementation of Pandas UDFs: Pandas Function APIs and Arrow Function APIs, as shown below:

!image-2023-03-29-11-40-44-318.png|width=576,height=225!

 

Spark Connect Python Client (SCPC), as a client and server interface for PySpark will eventually replace the legacy API of PySpark. Supporting PySpark UDFs is essential for Spark Connect to reach parity with the PySpark legacy API.

See design doc [here|https://docs.google.com/document/d/e/2PACX-1vRXF8nTdjwH0LbYyp3b6Zt6STEKWsvfKSO7_s4foOB-3zJ2h4_06JF147hUPlADJxZ_X22RFxgZ-fRS/pub].