av9ash/GitBugs huggingface dataset:
Bug: Support Hive 4.0 metastore
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/23 23:59
Affects Version/s: 4.0.0

Although Hive 4.0.0 is still beta I would like to work on this as Hive 4.0.0 will support support the pushdowns of partition column filters with VARCHAR/CHAR types.

For details please see HIVE-26661: Support partition filter for char and varchar types on Hive metastore
Bug: Make Yarn executor's bindAddress configurable
Status: Open | Priority: Major | Resolution: 
Created: 22/Sep/23 09:42
Affects Version/s: 3.5.0

An improvement has been made in SPARK-24203 that executor's bind address is configurable. Unfortunately this configuration hasn't implemented in Yarn.

When Yarn cluster is deployed in in Kubernetes, it is preferable to bind the executor to loopback interface or all interface. This Jira is to allow Yarn to bind the executor to either pod IP or loopbak interface or all interface to allow mesh integration like Istio with the cluster.

Another linked Jira explained *Allowing binding to all IPs* very well SPARK-42411
Bug: Optimize the processing speed of `BinaryArithmetic#dataType` when processing multi-column data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Sep/23 13:23
Affects Version/s: 3.4.0

Since `BinaryArithmetic#dataType` will recursively process the datatype of each node, the driver will be very slow when multiple columns are processed.

For example, the following code:
{code:java}
```
    import spark.implicits._
    import scala.util.Random
    import org.apache.spark.sql.functions.sum
    import org.apache.spark.sql.types.{StructType, StructField, IntegerType}
    val N = 30
    val M = 100
    val columns = Seq.fill(N)(Random.alphanumeric.take(8).mkString)
    val data = Seq.fill(M)(Seq.fill(N)(Random.nextInt(16) - 5))
    val schema = StructType(columns.map(StructField(_, IntegerType)))
    val rdd = spark.sparkContext.parallelize(data.map(Row.fromSeq(_)))
    val df = spark.createDataFrame(rdd, schema)
    val colExprs = columns.map(sum(_))
    // gen a new column , and add the other 30 column
    df.withColumn("new_col_sum", expr(columns.mkString(" + ")))
```
{code}
 

This code will take a few minutes for the driver to execute in the spark3.4 version, but only takes a few seconds to execute in the spark3.2 version. Related issue: SPARK-39316
Bug: Mark connection as timedOut in TransportClient.close
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/23 23:43
Affects Version/s: 3.3.4

Avoid a race condition where a connection which is in the process of being closed could be returned by the TransportClientFactory only to be immediately closed and cause errors upon use

 

This doesn't happen much in practice but is observed more frequently as part of efforts to add SSL support
Bug: XML: StructType schema issue in pyspark connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/23 02:09
Affects Version/s: 4.0.0

The following PR added support for from_xml to pyspark.

https://github.com/apache/spark/pull/42938

 

However, StructType schema is resulting in schema parse error for pyspark connect. 

Filing a Jira to track this issue.
Bug: Upgrade snappy to 1.1.10.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/23 19:29
Affects Version/s: 3.5.1

Security Fix
Fixed SnappyInputStream so as not to allocate too large memory when decompressing data with an extremely large chunk size by @​tunnelshade (code change)
This does not affect users only using Snappy.compress/uncompress methods
Bug: Make PySpark compatible with PyArrow 13.0.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/23 05:18
Affects Version/s: 4.0.0

[https://github.com/apache/spark/actions/runs/6167186123/job/16738683872]

 
{code:java}
======================================================================
FAIL [0.095s]: test_from_to_pandas (pyspark.pandas.tests.data_type_ops.test_datetime_ops.DatetimeOpsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/testing/pandasutils.py", line 122, in _assert_pandas_equal
    assert_series_equal(
  File "/usr/local/lib/python3.9/dist-packages/pandas/_testing/asserters.py", line 931, in assert_series_equal
    assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
  File "/usr/local/lib/python3.9/dist-packages/pandas/_testing/asserters.py", line 415, in assert_attr_equal
    raise_assert_detail(obj, msg, left_attr, right_attr)
  File "/usr/local/lib/python3.9/dist-packages/pandas/_testing/asserters.py", line 599, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Attributes of Series are different

Attribute "dtype" are different
[left]:  datetime64[ns]
[right]: datetime64[us] {code}
Bug: PythonWorkerFactory create gets stuck in case of worker failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/23 04:00
Affects Version/s: 4.0.0

`createSimpleWorker()` method in `PythonWorkerFactory` does not return if the worker fails to connect back to the server. 



This is because it calls accept() without a timeout. If the worker does not connect back, accept() waits forever. 

See PR for the fix.
Bug: Maven test `SparkConnectProtoSuite` failed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/23 12:31
Affects Version/s: 4.0.0

 

build/mvn clean install -pl connector/connect/server -am -DskipTests

mvn test -pl connector/connect/server 

 
{code:java}
- Test observe *** FAILED ***
  == FAIL: Plans do not match ===
  !CollectMetrics my_metric, [min(id#0) AS min_val#0, max(id#0) AS max_val#0, sum(id#0) AS sum(id)#0L], 0   CollectMetrics my_metric, [min(id#0) AS min_val#0, max(id#0) AS max_val#0, sum(id#0) AS sum(id)#0L], 53
   +- LocalRelation <empty>, [id#0, name#0]                                                                 +- LocalRelation <empty>, [id#0, name#0] (PlanTest.scala:179) {code}
Bug: Adjust sorter for Aggregate after SMJ
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/23 06:53
Affects Version/s: 4.0.0

If there is an aggregate operator after the SMJ and the grouping expressions of aggregate operator contain all the join keys of the streamed side, we can add a sorter on the streamed side of the SMJ, so that the aggregate can be convert to a SortAggregate which will be faster than HashAggregate.

For example, with table t1(a, b, c) and t2(x, y, z):
{code:java}
SELECT a, b, sum(c)
FROM t1
JOIN t2
ON t1.b = t2.y
GROUP BY a, b
{code}

Before this PR:
{code:java}
Scan(t1)                Scan(t2)
    |                      |
    |                      |
Exchange 1           Exchange 2
     \                     /
   Sort(t1.b)        Sort(t2.y) 
         \             /
       SMJ (t1.b = t2.y)
                |
                |
         HashAggregate
{code}
We can change Sort(t1.b)  to Sort(t1.b, t1.a) in the left side of the SMJ, and then the following aggregate could be convert to SortAggregate, which will be faster.

{code:java}
Scan(t1)                Scan(t2)
    |                      |
    |                      |
Exchange 1           Exchange 2
     \                    /
 Sort(t1.b, t1.a)   Sort(t2.y) 
         \             /
      SMJ (t1.b = t2.y)
                |
                |
          SortAggregate
{code}
Benchmark result
{code:java}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Aggregate after SMJ:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Hash aggregate after SMJ                          50508          50667         225          0.4        2408.4       1.0X
Sort aggregate after SMJ                          27556          27734         252          0.8        1314.0       1.8X
{code}
Bug: The return status is incorrect in standalone mode
Status: Open | Priority: Major | Resolution: 
Created: 24/Sep/23 01:53
Affects Version/s: 3.4.0

 When a job is submitted in standalone mode using spark launch, sparkLancher returns a successful execution and then a failed execution. Examples are as follows

log: Spark App Id [app-20230922160022-0006] State Changed.  State [FINISHED]
23/09/22 16:01:09 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/09/22 16:01:10 INFO MemoryStore: MemoryStore cleared
23/09/22 16:01:10 INFO BlockManager: BlockManager stopped
23/09/22 16:01:10 INFO BlockManagerMaster: BlockManagerMaster stopped
23/09/22 16:01:10 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/09/22 16:01:10 INFO SparkContext: Successfully stopped SparkContext
23/09/22 16:01:10 INFO ShutdownHookManager: Shutdown hook called
23/09/22 16:01:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-e015933f-a220-45a8-9d73-650b8bd8a337
23/09/22 16:01:10 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-8879f1d8-1f21-4b52-bca1-d3c66af6f754
23/09/22 16:01:11 INFO log: Spark App Id [app-20230922160022-0006] State Changed.  State [FAILED]
Bug: Eliminate foldable window partitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/23 08:12
Affects Version/s: 3.5.0

Foldable partition is redundant, remove it not only can simplify plan, but some rules can also take effect when the partitions are all foldable, such as `LimitPushDownThroughWindow{{{}`{}}}.
Bug: Release cast from attribute in filter to support predicate push down
Status: Open | Priority: Major | Resolution: 
Created: 18/Sep/23 09:47
Affects Version/s: 3.5.0

When a {{cast clause}} was used in a column in a filter, it will not be able to push down. We can remove the cast from attribute side to support predicate push down if the cast not change the precision or range.

Test query:
{code:sql}
-- dt is string type
explain select * from wechar_tbl where cast(dt as date) = date_sub(current_date(), 1);
{code}

Before this patch:
{code:bash}
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet default.wechar_tbl[id#5,name#6,dt#7] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(0 paths)[], PartitionFilters: [isnotnull(dt#7), (cast(dt#7 as date) = 2023-09-17)], PushedFilters: [], ReadSchema: struct<id:int,name:string>
{code}
*cast(dt#7 as date) = 2023-09-17 can not be push down in partition filter*

After this patch:
{code:bash}
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet default.wechar_tbl[id#62,name#63,dt#64] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[], PartitionFilters: [isnotnull(dt#64), (dt#64 = 2023-09-17)], PushedFilters: [], ReadSchema: struct<id:int,name:string>
{code}
*(dt#64 = 2023-09-17) can be push down in partition filter*
Bug: Add pyspark "ml-connect" extras dependencies
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/23 08:22
Affects Version/s: 4.0.0

Add pyspark "ml-connect" extras dependencies

dependencies includes:

```

torch # used in model distributed training

torcheval # used in metric evaluation

scikit-learn # used in exporting feature transformer models.

```
Bug: Support keyword columns on filters that interact with HMS
Status: Open | Priority: Major | Resolution: 
Created: 07/Sep/23 16:18
Affects Version/s: 3.4.1

Recently, https://issues.apache.org/jira/browse/HIVE-27665 was pushed on Hive. This will allow HMS to handle columns that are surrounded by backticks in filters.  An example of a customer who hit this problem had a filter in Spark like this:

where date='2015-01-06'

This didn't work because the word "date" is a keyword.  In order for the customer to work, the where clause should be changed to this:

where `date`='2015-01-06'

Spark strips out the backticks before passing the filter to HMS.  We need to no longer strip the backticks as a configurable flag.
Bug: Do not transpose windows if they conflict on ORDER BY / PROJECT clauses
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/23 18:59
Affects Version/s: 3.4.1

TransposeWindows rule reorders parent and child window functions. Currently it incorrectly reorders the window functions in cases where the top window function orders by on the result of the bottom window function, e.g. {{sum1}} in the following example:

{{SELECT ROW_NUMBER() OVER (PARTITION BY C ORDER BY sum1)  FROM (SELECT ROW_NUMBER() OVER (PARTITION BY C) as sum1 FROM T) }}
Bug: spark using --proxy-user GSS init failed when `hive.metastore.token.signature` not empty
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/23 00:12
Affects Version/s: 3.4.0

In spark, we can using --proxy-user to proxy the other user in kerberos env. But we will make  GSS init failed exception when connect to hive metastore and `hive.metastore.token.signature` not empty.
{code:java}
```
spark-sql  --conf spark.driver.extraClassPath=/home/hive/conf   --proxy-user test_user 
```{code}

if we set conf in `hive-site.xml`
{code:java}
```
    <property>
       <name>hive.metastore.token.signature</name>
       <value>spark_delegation_token</value>
    </property>
```{code}
we will get 

 
{code:java}
```
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
    at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:95)
    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
    at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:38)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:478)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245)
```{code}
Bug: Encoders.bean does no longer work with read-only properties
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/23 14:42
Affects Version/s: 3.4.1

Since Spark 3.4.x an exception is thrown when Encoders.bean is called providing a bean having read-only properties, such as:

 
{code:java}
public static class ReadOnlyPropertyBean implements Serializable {
    public boolean isEmpty() {
      return true;
    }
} {code}
 

 
Encoders.bean(ReadOnlyPropertyBean.class) will throw:
{code:java}
java.util.NoSuchElementException: None.get
        at scala.None$.get(Option.scala:529)
        at scala.None$.get(Option.scala:527)
        at org.apache.spark.sql.catalyst.ScalaReflection$.$anonfun$deserializerFor$8(ScalaReflection.scala:359)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at org.apache.spark.sql.catalyst.ScalaReflection$.deserializerFor(ScalaReflection.scala:348)
        at org.apache.spark.sql.catalyst.ScalaReflection$.deserializerFor(ScalaReflection.scala:183)
        at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:56)
        at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.javaBean(ExpressionEncoder.scala:62)
        at org.apache.spark.sql.Encoders$.bean(Encoders.scala:179)
        at org.apache.spark.sql.Encoders.bean(Encoders.scala) {code}
This problem is described also in [link Encoders.bean doesn't work anymore on a Java POJO, with Spark 3.4.0|https://stackoverflow.com/questions/76036349/encoders-bean-doesnt-work-anymore-on-a-java-pojo-with-spark-3-4-0]
Bug: Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/23 08:46
Affects Version/s: 3.4.0

If you find it convenient, you might clone the [https://gitlab.com/territoirevif/minimal-tests-spark-issue] project (that does many operations around cities, local authorities and accounting with open data) where I've extracted from my work what's necessary to make a set of 35 tests that run correctly with Spark 3.3.x, and show the troubles encountered with 3.4.x and 3.5.x.

 

It is working well with Spark 3.2.x, 3.3.x. But as soon as I selec{*}t Spark 3.4.x{*}, where the encoder seems to have deeply changed, the encoder fails with two problems:

 

*1)* It throws *java.util.NoSuchElementException: None.get* messages everywhere.

Asking over the Internet, I wasn't alone facing this problem. Reading it, you'll see that I've attempted a debug but my Scala skills are low.

[https://stackoverflow.com/questions/76036349/encoders-bean-doesnt-work-anymore-on-a-java-pojo-with-spark-3-4-0]

{color:#172b4d}by the way, if possible, the encoder and decoder functions should forward a parameter as soon as the name of the field being handled is known, and then all the long of their process, so that when the encoder is at any point where it has to throw an exception, it knows the field it is handling in its specific call and can send a message like:{color}
{color:#00875a}_java.util.NoSuchElementException: None.get when encoding [the method or field it was targeting]_{color}

 

*2)* *Not found an encoder of the type RS to Spark SQL internal representation.* Consider to change the input type to one of supported at (...)
Or : Not found an encoder of the type *OMI_ID* to Spark SQL internal representation (...)

 
where *RS* and *OMI_ID* are generic types.
This is strange.
[https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge]

 

*3)* When I switch to the *Spark 3.5.0* version, the same problems remain, but another add itself to the list:
"{*}Only expression encoders are supported for now{*}" on what was accepted and working before.
Bug: Support spark.master.rest.host
Status: Closed | Priority: Major | Resolution: Won't Fix
Created: 06/Sep/23 23:20
Affects Version/s: 4.0.0

To allow Spark REST server to choose the host address to listen.

For example, in K8s environment, the AS-IS Spark REST Server only listens the podIP and doesn't listen 127.0.0.1. As a result, K8s post-forward is not applicable.
Bug: AddArtifacts should give proper error messages if it fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/23 14:46
Affects Version/s: 3.5.0

I've been trying to do some testing of udf's using code in other module, so that  AddArtifact is necessary.

 

I got the following error:

 

 
{code:java}
Traceback (most recent call last):
  File "/Users/alice.sayutina/db-connect-playground/udf2.py", line 5, in <module>
    spark.addArtifacts("udf2_support.py", pyfile=True)
  File "/Users/alice.sayutina/db-connect-venv/lib/python3.10/site-packages/pyspark/sql/connect/session.py", line 744, in addArtifacts
    self._client.add_artifacts(*path, pyfile=pyfile, archive=archive, file=file)
  File "/Users/alice.sayutina/db-connect-venv/lib/python3.10/site-packages/pyspark/sql/connect/client/core.py", line 1582, in add_artifacts
    self._artifact_manager.add_artifacts(*path, pyfile=pyfile, archive=archive, file=file)
  File "/Users/alice.sayutina/db-connect-venv/lib/python3.10/site-packages/pyspark/sql/connect/client/artifact.py", line 283, in add_artifacts
    self._request_add_artifacts(requests)
  File "/Users/alice.sayutina/db-connect-venv/lib/python3.10/site-packages/pyspark/sql/connect/client/artifact.py", line 259, in _request_add_artifacts
    response: proto.AddArtifactsResponse = self._retrieve_responses(requests)
  File "/Users/alice.sayutina/db-connect-venv/lib/python3.10/site-packages/pyspark/sql/connect/client/artifact.py", line 256, in _retrieve_responses
    return self._stub.AddArtifacts(requests, metadata=self._metadata)
  File "/Users/alice.sayutina/db-connect-venv/lib/python3.10/site-packages/grpc/_channel.py", line 1246, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/alice.sayutina/db-connect-venv/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Exception iterating requests!"
        debug_error_string = "None"
{code}
 

Which doesn't give any clue about what happens.

Only after noticeable investigation I found the problem: I'm specifying the wrong path and the artifact fails to upload. Specifically what happens is that ArtifactManager doesn't read the file immediately, but rather creates iterator object which will incrementally generate requests to send. This iterator is passed to grpc's stream_unary to consume and actually send, and while grpc catches the error (see above), it suppresses the underlying exception.

I think we should improve pyspark user experience. One of the possible ways to fix this is to wrap ArtifactsManager._create_requests with an iterator wrapper which would log the throwable into spark connect logger so that user would see something like below at least when the debug mode is on.

 
{code:java}
FileNotFoundError: [Errno 2] No such file or directory: '/Users/alice.sayutina/udf2_support.py' {code}
Bug: Fix non-deterministic seeded Dataset APIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Sep/23 12:00
Affects Version/s: 4.0.0

If we run the following example the result is the expected equal 2 columns:

{noformat}
val c = rand()
df.select(c, c)

+--------------------------+--------------------------+
|rand(-4522010140232537566)|rand(-4522010140232537566)|
+--------------------------+--------------------------+
|        0.4520819282997137|        0.4520819282997137|
+--------------------------+--------------------------+
{noformat}

 
But if we run use other similar APIs their result is incorrect:

{noformat}
val r1 = random()
val r2 = uuid()
val r3 = shuffle(col("x"))
val x = df.select(r1, r1, r2, r2, r3, r3)

+------------------+------------------+--------------------+--------------------+----------+----------+
|            rand()|            rand()|              uuid()|              uuid()|shuffle(x)|shuffle(x)|
+------------------+------------------+--------------------+--------------------+----------+----------+
|0.7407604956381952|0.7957319451135009|e55bc4b0-74e6-4b0...|a587163b-d06b-4bb...| [1, 2, 3]| [2, 1, 3]|
+------------------+------------------+--------------------+--------------------+----------+----------+
{noformat}
Bug: Correctness issue when persisting using StorageLevel.NONE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/23 09:31
Affects Version/s: 3.5.0

When using spark 3.5.0 this code
{code:java}
import org.apache.spark.storage.StorageLevel
spark.createDataset(Seq(1,2,3)).persist(StorageLevel.NONE).count() {code}
incorrectly returns 0.