av9ash/GitBugs huggingface dataset:
Bug: No overload variant of "pandas_udf" matches argument type "str"
Status: Open | Priority: Major | Resolution: 
Created: 19/Apr/23 12:29
Affects Version/s: 3.2.4

h2. Issue

Users who have mypy enabled in their IDE or CI environment face very verbose error messages when using the {{pandas_udf}} function in PySpark. The current typing of the {{pandas_udf}} function seems to be causing these issues. As a workaround, the official documentation provides examples that use {{{}# type: ignore[call-overload]{}}}, but this is not an ideal solution.
h2. Example

Here's a code snippet taken from [docs|https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html#pandas-udfs-a-k-a-vectorized-udfs] that triggers the error when mypy is enabled:
{code:python}
from pyspark.sql.functions import pandas_udf
import pandas as pd

@pandas_udf("col1 string, col2 long")
def func(s1: pd.Series, s2: pd.Series, s3: pd.DataFrame) -> pd.DataFrame:
    s3['col2'] = s1 + s2.str.len()
    return s3 {code}
Running mypy on this code results in a long and verbose error message, which makes it difficult for users to understand the actual issue and how to resolve it.
h2. Proposed Solution

We kindly request the PySpark development team to review and improve the typing for the {{pandas_udf}} function to prevent these verbose error messages from appearing. This improvement will help users who have mypy enabled in their development environments to have a better experience when using PySpark.

Furthermore, we suggest updating the official documentation to provide better examples that do not rely on {{# type: ignore[call-overload]}} to suppress these errors.
h2. Impact

By addressing this issue, users of PySpark with mypy enabled in their development environment will be able to write and verify their code more efficiently, without being overwhelmed by verbose error messages. This will lead to a more enjoyable and productive experience when working with PySpark and pandas UDFs.
Bug: Improve ArrayInsert if the position is foldable and equals to zero.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/23 03:37
Affects Version/s: 3.5.0

We want make array_prepend reuse the implementation of array_insert, but it seems a bit performance worse if the position is foldable and equals to zero.
The reason is that always do the check for position is negative or positive, and the code is too long. Too long code will lead to JIT failed.
Bug: Expose host and bearer tokens from the spark connect client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/23 09:37
Affects Version/s: 3.4.0

The `SparkConnectClient` class takes in a connection string to connect with the spark connect service.

 

As part of setting up the connection, it parses the connection string. Expose the parsed host and bearer tokens as part of the class, so they may be accessed by consumers.
Bug: Separate test into `pyspark-conenct-pandas` and `pyspark-connect-pandas-slow`
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Apr/23 06:27
Affects Version/s: 3.5.0

The test `pyspark-connect` takes 2~3 hours due to recently added pandas API on Spark tests, so we'd better to separate pandas API on Spark tests into different test module to reduce the overhead onto `pyspark-connect`.
Bug: An exception occurred while hive table join tidb table
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Apr/23 11:36
Affects Version/s: 3.2.3

When executing a query of a hive partition table (big one) inner join a tidb table(small one), the hive partition table is auto broadcasted, which leads an error.
The query is somelike
 {{select hive_table.col1,tidb_table.col2 from hive_table inner join tidb_table on hive_table.col2=tidb_table.col3 where ...}}
== Physical Plan ==
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [... 109 more fields]
+- Generate HiveGenericUDTF#udf.json.JsonExtractValueUDTF(xxx), [, ... 101 more fields], false, [...]
+- Project [, ... 102 more fields]
+- BroadcastHashJoin [xxx#94], [xxxx#475], Inner, BuildRight, false
:- TiKV CoprocessorRDD\{[table: xxx] TableReader, Columns: xxxx(): { TableRangeScan: { RangeFilter: [], Range: [([t\200\000\000\000\000\000\004\253_r\200\000\000\000\000\000\000\000], [t\200\000\000\000\000\000\004\253_s\000\000\000\000\000\000\000\000])([t\200\000\000\000\000\000\004\253_r\000\000\000\000\000\000\000\000], [t\200\000\000\000\000\000\004\253_r\200\000\000\000\000\000\000\000])] } }, startTs: 440854942292115639} EstimatedCount:20837
+- BroadcastExchange HashedRelationBroadcastMode(List(input[107, string, false]),false), [plan_id=32]
+- Filter isnotnull(xxx#475)
+- Scan hive xx.xxxxxxxx [, ... 100 more fields], HiveTableRelation [{{{}xx{}}}.{{{}xxx{}}}, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, Data Cols: [..., Partition Cols: [[#520|https://github.com/pingcap/tispark/issues/520], [#521|https://github.com/pingcap/tispark/issues/521], [#522|https://github.com/pingcap/tispark/issues/522], [#523|https://github.com/pingcap/tispark/pull/523]], Pruned Partitions: [(, , , )]], [isnotnull(), (), (xx = xx)]

Here I got some log info maybe helpful.
The {{plan.stats.sizeInBytes}} of the LogicalPlan of the hive table is too small and the {{plan.stats.sizeInBytes}} of LogicalPlan of the tidb table is too big.
The stats of the LogicalPlans of the two seems reversed.

*Spark and TiSpark version info*
Spark 3.2.3
TiSpark 3.1.2(with a profile of spark-3.2)
*Additional context*
Bug: Spark overwrites existing FILES/ARCHIVES/... settings instead of merging them
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/23 13:12
Affects Version/s: 3.3.2

https://issues.apache.org/jira/browse/SPARK-27575 fixes --files/–archives merging issues for spark on yarn mode.

 

However the overwrite logic remains in other resource managers, such as standalone, mesos and k8s. 

I'd like to propose that the merging behavior should be consistent across resource managers.
Bug: Throw error in planning phase when the query redefines watermark for a stream
Status: Open | Priority: Major | Resolution: 
Created: 17/Apr/23 07:36
Affects Version/s: 3.5.0

With SPARK-42376, we started to disallow redefinition of watermark in a single stream. But the check is actually performed in watermark propagator, which does not throw an exception in the batch 0.

Since we tend to give a fast feedback for improper usage, throwing exception should also happen in planning phase, ideally in UnsupportedOperationChecker.

In addition, we don't seem to document this restriction. Also update the structured streaming doc.
Bug: Set upperbound of pandas version in binder integrations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/23 00:48
Affects Version/s: 3.4.0

{code}
df.toPandas
{code}

fails with

{code}

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[14], line 1
----> 1 df.toPandas()

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/sql/pandas/conversion.py:251, in PandasConversionMixin.toPandas(self)
    248 should_check_timedelta = is_timedelta64_dtype(t) and len(pdf) == 0
    250 if (t is not None and not is_timedelta64_dtype(t)) or should_check_timedelta:
--> 251     series = series.astype(t, copy=False)
    253 with catch_warnings():
    254     from pandas.errors import PerformanceWarning

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/generic.py:6324, in NDFrame.astype(self, dtype, copy, errors)
   6317     results = [
   6318         self.iloc[:, i].astype(dtype, copy=copy)
   6319         for i in range(len(self.columns))
   6320     ]
   6322 else:
   6323     # else, only a single dtype is given
-> 6324     new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
   6325     return self._constructor(new_data).__finalize__(self, method="astype")
   6327 # GH 33113: handle empty frame or series

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/internals/managers.py:451, in BaseBlockManager.astype(self, dtype, copy, errors)
    448 elif using_copy_on_write():
    449     copy = False
--> 451 return self.apply(
    452     "astype",
    453     dtype=dtype,
    454     copy=copy,
    455     errors=errors,
    456     using_cow=using_copy_on_write(),
    457 )

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pandas/core/internals/managers.py:352, in BaseBlockManager.apply(self, f, align_keys, **kwargs)
    350         applied = b.apply(f, **kwargs)
    351     else:
--> 352         applied = getattr(b, f)(**kwargs)
    353     result_blocks = extend_blocks(applied, result_blocks)
    355 out = type(self).from_blocks(result_blocks, self.axes)
{code}
Bug: `Class.getCanonicalName` return null for anonymous class on JDK15+, impacting function registry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/23 22:19
Affects Version/s: 3.4.0

On JDK15+, lambda and method references are implemented using hidden classes ([https://openjdk.org/jeps/371)] According to the JEP, 
{quote}{{Class::getCanonicalName}} returns {{{}null{}}}, indicating the hidden class has no canonical name. (Note that the {{Class}} object for an anonymous class in the Java language has the same behavior.)
{quote}

This means [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L53] will always be null.

 

This can be fixed by replacing `getCanonicalName` with `getName`
Bug: Ignore generated Java files in checkstyle
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/23 11:36
Affects Version/s: 3.4.1

Files such as {{.../spark/core/target/scala-2.12/src_managed/main/org/apache/spark/status/protobuf/StoreTypes.java}} are checked in checkstyle. We shouldn't check them in the linter.
Bug: Fix construct aggregate expressions by replacing grouping functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/23 14:20
Affects Version/s: 3.5.0


{code:sql}
CREATE TEMPORARY VIEW grouping AS SELECT * FROM VALUES
  ("1", "2", "3", 1),
  ("4", "5", "6", 1),
  ("7", "8", "9", 1)
  as grouping(a, b, c, d);
{code}

{noformat}
spark-sql (default)> SELECT CASE WHEN a IS NULL THEN count(b) WHEN b IS NULL THEN count(c) END
                   > FROM grouping
                   > GROUP BY GROUPING SETS (a, b, c);
[MISSING_AGGREGATION] The non-aggregating expression "b" is based on columns which are not participating in the GROUP BY clause.
{noformat}
Bug: pandas udf/udaf not support partial merge
Status: Open | Priority: Major | Resolution: 
Created: 13/Apr/23 13:29
Affects Version/s: 3.3.2

hi community:
I use pandas udf in pysaprk,when I apply udf on grouped dataframe,it cost too many memory,I think it because pandas udaf not support partial merge
 
code like:
spark_df.groupby('a').applyinpandas(func,schema)
Bug: Fix Cast.forceNullable
Status: Open | Priority: Major | Resolution: 
Created: 13/Apr/23 02:48
Affects Version/s: 3.5.0

https://github.com/apache/spark/blob/ac105ccebf5f144f2d506cbe102c362a195afa9a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L381-L399

1. It should include TimestampNTZType,AnsiIntervalType.
2. ArrayType/MapType/StructType to other data type seems should be true.
Bug:  JavaRDD.saveAsTextFile Directory Creation issue using Spark 3.3.2, with hadoop3
Status: Open | Priority: Major | Resolution: 
Created: 12/Apr/23 16:30
Affects Version/s: 3.3.2

We are using Spark 3.3.2, with hadoop3 in Java Appliaction .

While Saving file using JavaRDD.saveAsTextFile(DIR_NAME/Sample.txt).

 

Spark is creating DIR_NAME with the same user by which Application is Running ex vs.

However we have provide another user to create the DIR_NAME like below in our Spark code .

UserGroupInformation theSparkUser= UserGroupInformation.createRemoteUser("spark");
UserGroupInformation.setLoginUser(theSparkUser);

 

Below is the dir structure created by the Application.

drwxrwxrwx+ 2  vs vs 6 Apr 12 17:09  main

 

We want Dir should be created by the user that we have provided, Like below

drwxrwxrwx+ 2  spark spark  6 Apr 12 17:09  main

 

When we are verifying the user by printing its showing same user as we pass in UserGroupInformation  both below printing same user "spark"

 

UserGroupInformation.getLoginUser().getUserName());   --> printing user as "spark"
UserGroupInformation.getCurrentUser().getUserName());  --> printing user as "spark"
Bug: Support the CBC mode by aes_encrypt()/aes_decrypt()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/23 15:24
Affects Version/s: 3.5.0

Implement the GCM mode in CBC - aes_encrypt() and aes_decrypt(). Use *AES/CBC/PKCS5Padding* of the Java Cryptographic Extension (JCE) framework. Output should be compatible with OpenSSL.
Bug: Support custom fields in the file source _metadata column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/23 15:36
Affects Version/s: 3.4.0

Today, the schema of the file source _metadata column depends on the file format (e.g. parquet file format supports {{{}_metadata.row_index{}}}) but this is hard-wired into the {{FileFormat}} itself. Not only is this an ugly design, it also prevents custom file formats from adding their own fields to the {{_metadata}} column.
Bug: Upgrade zstd-jni to 1.5.5-1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/23 09:02
Affects Version/s: 3.5.0

* [luben/zstd-jni@{{{}v1.5.4-2...v1.5.5-1{}}}|https://github.com/luben/zstd-jni/compare/v1.5.4-2...v1.5.5-1]
Bug: Add error class SQL_CONF_NOT_FOUND
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/23 08:20
Affects Version/s: 3.5.0

Use the error class when getting a non-existing SQL config. Currently the error message is not user-friendly:
{code:java}
spark.conf.get("some.conf") 

java.util.NoSuchElementException: some.conf
 at org.apache.spark.sql.errors.QueryExecutionErrors$.noSuchElementExceptionError(QueryExecutionErrors.scala:2154)
 at org.apache.spark.sql.internal.SQLConf.$anonfun$getConfString$3(SQLConf.scala:4947)
...{code}
Bug: Avoid task retries due to AssertNotNull checks
Status: Open | Priority: Major | Resolution: 
Created: 04/Apr/23 20:42
Affects Version/s: 3.5.0

As discussed [here|https://github.com/apache/spark/pull/40655#discussion_r1156693696], tasks that failed because of exceptions generated by {{AssertNotNull}} should not be retried.
Bug: Improve the error message of UNRECOGNIZED_SQL_TYPE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/23 02:49
Affects Version/s: 3.5.0

UNRECOGNIZED_SQL_TYPE prints the jdbc type id in the error message currently. This is difficult for spark users to understand the meaning of this kind of error, especially when the type id is from a vendor extension.

For example, 
{code:java}
 org.apache.spark.SparkSQLException: Unrecognized SQL type -102{code}
Bug: Support overloading UDF
Status: Open | Priority: Major | Resolution: 
Created: 11/Apr/23 02:21
Affects Version/s: 3.0.2

It seems that Spark SQL does not support overloading UDF for a long while. If we register two functions with the same name, Spark complains "The function is replaced a previously registered function". The solution is to either enhancing the org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry class to support multiple functions with the same name, or enable users to extend and use their own FunctionRegistry class. Should you have any comment, please kindly let me know.
Bug: Add bloom filter details in spark history server plans/SVGs
Status: Open | Priority: Major | Resolution: 
Created: 10/Apr/23 07:06
Affects Version/s: 3.3.2

Spark bloom filter can be enabled via "spark.sql.optimizer.runtimeFilter.semiJoinReduction.enabled=true  and spark.sql.optimizer.runtime.bloomFilter.enabled=true".

Spark history server's SVG doesn't render the bloom filter details; It will be good to include this detail in the plan. (as of now, it shows up explain plan's text output).
Bug: Set job description for tpcds queries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/23 08:51
Affects Version/s: 3.1.2

 

When using Spark's TPCDSQueryBenchmark to run tpcds, the spark ui does not display the sql information

!https://user-images.githubusercontent.com/94670132/230567550-9bb2842c-aecc-41a5-acb6-0ff8ea765df1.png|width=1694,height=523!
Bug: Add the function without constant parameters of `SessionState#executePlan`
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 08/Apr/23 13:41
Affects Version/s: 3.2.3

 
{code:java}
```
df = spark.sql("select 1") 
catalyst_plan = df._jdf.queryExecution().logical()
print('catalyst_plan: ', catalyst_plan)
df_size = spark._jsparkSession.sessionState().executePlan(catalyst_plan)
```
{code}
 

 

will get exception beacause py4j not support default value of scala.

 

[https://github.com/apache/spark/blob/d8b720a579a456aec9c7f843d7feaa1454ebf9d4/sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala#L124]

 
{code:java}
```
def executePlan(
plan: LogicalPlan,
mode: CommandExecutionMode.Value = CommandExecutionMode.ALL): QueryExecution =
createQueryExecution(plan, mode)
```
{code}
 

 

Will get Exception:

 

 
{code:java}
```
py4j.protocol.Py4JError: An error occurred while calling o87.executePlan. Trace:
py4j.Py4JException: Method executePlan([class org.apache.spark.sql.catalyst.plans.logical.Project]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:321)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:329)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.lang.Thread.run(Thread.java:748)
```
{code}
Bug: Error class resource file in Kafka connector is misplaced
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/23 10:17
Affects Version/s: 3.4.0

SPARK-41387 adopted error class framework in Kafka connector. Since it's a connector, the error class file is intentionally put to the Kafka connector module instead of being added to the Spark's central error class file.

Unfortunately I've figured out somehow that the place is wrong. It should be placed to the resources directory in source, not test. The problem cannot be captured in test suite as it's available in test artifact, but I can trigger the problem via adding Kafka connector jar into classpath and initialize KafkaExceptions object.

Hopefully the blast radius of the problem is trivial as Kafka connector uses error class only for assertions which should not be triggered unless some accident happens e.g. topic is deleted and recreated while the streaming query is running with Trigger.Available.
Bug: Use `sbt-eclipse` instead of `sbteclipse-plugin`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/23 18:57
Affects Version/s: 3.2.3

Since SPARK-34959, Apache Spark 3.2+ uses SBT 1.5.0.

And, SBT 1.4+ can use `set-eclipse` instead of old `sbteclipse-plugin`.
- https://github.com/sbt/sbt-eclipse/releases
Bug: Introduce dropDuplicatesWithinWatermark in PySpark
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 04/Apr/23 06:56
Affects Version/s: 3.5.0

This is a placeholder to ensure we push PySpark API as well when we introduce dropDuplicatesWithinWatermark in Scala/Java API. When the implementation is merged in Scala/Java side, adding API layer to PySpark would be straightforward, so intentionally separated this out.
Bug: BlockStoreClient getHostLocalDirs RPC supports IOException retry
Status: Open | Priority: Minor | Resolution: 
Created: 27/Apr/23 03:42
Affects Version/s: 3.0.0

BlockStoreClient#getHostLocalDirs RPC did not retry when IOexception occurred, and then FetchFailedException was thrown.

 
{code:java}
23/04/24 01:24:55,158 [shuffle-client-7-1] WARN ExternalBlockStoreClient: Error while trying to get the host local dirs for [148]
23/04/24 01:24:55,158 [shuffle-client-7-1] ERROR ShuffleBlockFetcherIterator: Error occurred while fetching host local blocks
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:745) {code}
Bug: Docker images are missing passwd entry for UID 185
Status: Open | Priority: Minor | Resolution: 
Created: 17/Apr/23 17:46
Affects Version/s: 3.4.0

Currently, the official Spark docker images run as the UID {{{}185{}}}, but no corresponding entry exists in {{{}/etc/passwd{}}}. This causes [issues|https://stackoverflow.com/questions/41864985/hadoop-ioexception-failure-to-login] when libraries try to fetch the current unix username.
Bug: Assign a name to the error class _LEGACY_ERROR_TEMP_202[3,4,5]
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/23 09:38
Affects Version/s: 3.5.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_2023* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]