av9ash/GitBugs huggingface dataset:
Bug: Local Property Propagation to Subquery Broadcast Exec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/23 16:49
Affects Version/s: 3.4.0

https://issues.apache.org/jira/browse/SPARK-32748 was opened and then I believe mistakenly reverted to address this issue. The claim was local properties propagation in SubqueryBroadcastExec to the dynamic pruning thread is not necessary because they will be propagated by broadcast threads anyways. However, in a scenario where the dynamic pruning thread is first to initialize the broadcast relation future, the local properties will not be propagated correctly. This is because the local properties being propagated to the broadcast threads would already be incorrect.
I do not have a good way of reproducing this consistently because generally the SubqueryBroadcastExec is not the first to initialize the broadcast relation future, but by adding a Thread.sleep(1) into the doPrepare method of SubqueryBroadcastExec, the following test always fails.
{code:java}
withSQLConf(StaticSQLConf.SUBQUERY_BROADCAST_MAX_THREAD_THRESHOLD.key -> "1") {
  withTable("a", "b") {
    val confKey = "spark.sql.y"
    val confValue1 = UUID.randomUUID().toString()
    val confValue2 = UUID.randomUUID().toString()
    Seq((confValue1, "1")).toDF("key", "value")
      .write
      .format("parquet")
      .partitionBy("key")
      .mode("overwrite")
      .saveAsTable("a")
    val df1 = spark.table("a")

    def generateBroadcastDataFrame(confKey: String, confValue: String): Dataset[String] = {
      val df = spark.range(1).mapPartitions { _ =>
        Iterator(TaskContext.get.getLocalProperty(confKey))
      }.filter($"value".contains(confValue)).as("c")
      df.hint("broadcast")
    }

    // set local property and assert
    val df2 = generateBroadcastDataFrame(confKey, confValue1)
    spark.sparkContext.setLocalProperty(confKey, confValue1)
    val checkDF = df1.join(df2).where($"a.key" === $"c.value").select($"a.key", $"c.value")
    val checks = checkDF.collect()
    assert(checks.forall(_.toSeq == Seq(confValue1, confValue1)))

    // change local property and re-assert
    Seq((confValue2, "1")).toDF("key", "value")
      .write
      .format("parquet")
      .partitionBy("key")
      .mode("overwrite")
      .saveAsTable("b")
    val df3 = spark.table("b")
    val df4 = generateBroadcastDataFrame(confKey, confValue2)
    spark.sparkContext.setLocalProperty(confKey, confValue2)
    val checks2DF = df3.join(df4).where($"b.key" === $"c.value").select($"b.key", $"c.value")
    val checks2 = checks2DF.collect()
    assert(checks2.forall(_.toSeq == Seq(confValue2, confValue2)))
    assert(checks2.nonEmpty)
  }
} {code}
Bug: Make PySpark (pyspark-ml module) tests passing without any optional dependency
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/23 07:47
Affects Version/s: 4.0.0

{code}
./python/run-tests --python-executables=python3 --modules=pyspark-ml
...
Starting test(python3): pyspark.ml.tests.test_model_cache (temp output: /Users/hyukjin.kwon/workspace/forked/spark/python/target/f6f88c1e-0cb2-43e6-980e-47f1cdb9b463/python3__pyspark.ml.tests.test_model_cache__zij05l1u.log)
Traceback (most recent call last):
  File "/Users/hyukjin.kwon/miniconda3/envs/vanilla-3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/hyukjin.kwon/miniconda3/envs/vanilla-3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/hyukjin.kwon/workspace/forked/spark/python/pyspark/ml/tests/test_functions.py", line 18, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'
{code}
Bug: Fix SQLQueryTestSuite unable create table normally
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/23 06:24
Affects Version/s: 3.5.0

When we repeatedly execute SQLQueryTestSuite to generate the golden file, the warehouse file executed last time is not cleaned up (maybe killed when test not finish), resulting in an error result 

!image-2023-08-27-14-25-21-843.png!
Bug: Reuse ArrayInsert in ArrayAppend
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/23 13:36
Affects Version/s: 4.0.0

After the PR [https://github.com/apache/spark/pull/42564,] array_insert with -1 inserts an element at the end of the given array. So, array_append can be mapped to the array_insert() function.
Bug: Make PySpark (pyspark-connect module) tests passing without any dependency
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/23 07:11
Affects Version/s: 4.0.0

{code}
./python/run-tests --python-executables=python3 --modules=pyspark-connect
...
2c5289024a/python3__pyspark.sql.connect.window__nvbbzy7q.log)
Finished test(python3): pyspark.sql.connect.session (0s)
Traceback (most recent call last):
  File "/Users/hyukjin.kwon/workspace/forked/spark/python/pyspark/sql/pandas/utils.py", line 27, in require_minimum_pandas_version
    import pandas
ModuleNotFoundError: No module named 'pandas'
{code}
Bug: Make PySpark (pyspark-sql module) tests passing without any dependency
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/23 04:24
Affects Version/s: 4.0.0

{code}
./python/run-tests --python-executables=python3 --modules=pyspark-sql
Running PySpark tests. Output is in /.../spark/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python modules: ['pyspark-sql']
python3 python_implementation is CPython
python3 version is: Python 3.10.12
Starting test(python3): pyspark.sql.tests.pandas.test_pandas_grouped_map_with_state (temp output: /.../spark/python/target/8e530108-4d5e-46e4-88fb-8f0dfb7b47e2/python3__pyspark.sql.tests.pandas.test_pandas_grouped_map_with_state__jggatex7.log)
Starting test(python3): pyspark.sql.tests.pandas.test_pandas_grouped_map (temp output: /.../spark/python/target/3b6e9e5a-c479-408c-9365-8286330e8e7c/python3__pyspark.sql.tests.pandas.test_pandas_grouped_map__1lrovmur.log)
Starting test(python3): pyspark.sql.tests.pandas.test_pandas_cogrouped_map (temp output: /.../spark/python/target/68c7cf56-ed7a-453e-8d6d-3a0eb519d997/python3__pyspark.sql.tests.pandas.test_pandas_cogrouped_map__sw2875dr.log)
Starting test(python3): pyspark.sql.tests.pandas.test_pandas_map (temp output: /.../spark/python/target/90712186-a104-4491-ae0d-2b5ab973991b/python3__pyspark.sql.tests.pandas.test_pandas_map__ysp4911q.log)
Traceback (most recent call last):
  File "/.../miniconda3/envs/vanilla-3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/.../miniconda3/envs/vanilla-3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/.../workspace/forked/spark/python/pyspark/sql/tests/pandas/test_pandas_map.py", line 27, in <module>
    from pyspark.testing.sqlutils import (
  File "/.../workspace/forked/spark/python/pyspark/testing/__init__.py", line 19, in <module>
    from pyspark.testing.pandasutils import assertPandasOnSparkEqual
  File "/.../workspace/forked/spark/python/pyspark/testing/pandasutils.py", line 22, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'
{code}
Bug: Deterministic ApplyFunctionExpression should be foldable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/23 08:58
Affects Version/s: 3.4.1

Currently, ApplyFunctionExpression is unfoldable because inherits the default value from Expression.  However, it should be foldable for a deterministic ApplyFunctionExpression. This could help optimize the usage for V2 UDF applying to constant expressions.
Bug: toPandas() gives FutureWarning when containing columns of datatype timestamp
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 24/Aug/23 13:57
Affects Version/s: 3.4.1

When converting a Spark DataFrame into a pandas DataFrame, we get a FutureWarning when the DataFrame contains columns of type {{timestamp. }}

Reproducible example (that you can run locally):
{code:java}
from datetime import datetime

from pyspark.sql import SparkSession
import pandas as pd

spark = SparkSession.builder.getOrCreate()

df = pd.DataFrame({"foo": [datetime(2023, 1, 1), datetime(2023, 1, 1)]})

df_sp = spark.createDataFrame(df)

test = df_sp.toPandas()

// warning logs: 
/usr/local/lib/python3.10/site-packages/pyspark/sql/pandas/conversion.py:251: FutureWarning: Passing unit-less datetime64 dtype to .astype is deprecated and will raise in a future version. Pass 'datetime64[ns]' instead

{code}
Note that if we enable arrow (by setting {{{}config("spark.sql.execution.arrow.pyspark.enabled", "true"){}}}), this warning is gone! Although I admit that I have seen it popping up once with arrow enabled too, but I could not create a reproducible example out of that. 

 

Note that this basically means that I cannot use Spark with pandas 2.0 without Arrow enabled...
Bug: Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/23 08:00
Affects Version/s: 3.5.0

{code}
from pyspark.sql import functions as F
{code}

isn’t very Pythonic - it does not follow PEP 8, see  Package and Module Names (https://peps.python.org/pep-0008/#package-and-module-names).

{quote}
Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves
readability. Python packages should also have short, all-lowercase names, although the use of underscores
is discouraged.
{quote}

Therefore, the module’s alias should follow this. In practice, the uppercase is only used at the module/package
level constants in my experience, see also Constants (https://peps.python.org/pep-0008/#constants).

See also this stackoverflow comment (https://stackoverflow.com/questions/70458086/how-to-correctly-import-pyspark-sql-functions#comment129714058_70458115).
Bug: [CONNECT] jsonignore SparkListenerConnectOperationStarted.planRequest
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/23 23:54
Affects Version/s: 3.5.0

SparkListenerConnectOperationStarted was added as part of SPARK-43923.

SparkListenerConnectOperationStarted.planRequest cannot be serialized & deserialized from json as it has recursive objects.

Add @JsonIgnoreProperties(\{ "planRequest" }) to avoid failures
Bug: Add a button to download thread dump as a txt file in Spark UI 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/23 01:17
Affects Version/s: 4.0.0

Add a new button on the thread dump page of Spark Web Ui to download the thread dump in text format. 

Right now if a user wants to collect a thread dump for an executor or driver using Spark UI, the thread dump is displayed as an html page. User can click on expand all button and then save that html file or take screen shots. However for debugging purpose  it is always helpful to have the jstack output in raw text format without any html formatting and the process of saving the file should be simple as one click. In this Jira we propose to add a button on the Spark Ui page to be able to directly save or display the thread dump output in raw text format without html formatting.
Bug: array_insert() give wrong results for ngative index
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/23 00:14
Affects Version/s: 3.4.0

Unlike in Snowflake we decided that array_inert() is 1 based.
This means 1 is the first element in an array and -1 is the last. 
This matches the behavior of functions such as substr() and element_at().

 
{code:java}
> SELECT array_insert(array('a', 'b', 'c'), 1, 'z');
["z","a","b","c"]
> SELECT array_insert(array('a', 'b', 'c'), 0, 'z');
Error
> SELECT array_insert(array('a', 'b', 'c'), -1, 'z');
["a","b","c","z"]
> SELECT array_insert(array('a', 'b', 'c'), 5, 'z');
["a","b","c",NULL,"z"]
> SELECT array_insert(array('a', 'b', 'c'), -5, 'z');
["z",NULL,"a","b","c"]
> SELECT array_insert(array('a', 'b', 'c'), 2, cast(NULL AS STRING));
["a",NULL,"b","c"]
{code}
Bug: SparkSession.Builder should respect the options
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/23 02:30
Affects Version/s: 3.5.0

In connect session builder, we use {{config}} method to set options.
However, the options are actually ignored when we create a new session.

{code}
        def create(self) -> "SparkSession":
            has_channel_builder = self._channel_builder is not None
            has_spark_remote = "spark.remote" in self._options

            if has_channel_builder and has_spark_remote:
                raise ValueError(
                    "Only one of connection string or channelBuilder "
                    "can be used to create a new SparkSession."
                )

            if not has_channel_builder and not has_spark_remote:
                raise ValueError(
                    "Needs either connection string or channelBuilder to create a new SparkSession."
                )

            if has_channel_builder:
                assert self._channel_builder is not None
                session = SparkSession(connection=self._channel_builder)
            else:
                spark_remote = to_str(self._options.get("spark.remote"))
                assert spark_remote is not None
                session = SparkSession(connection=spark_remote)

            SparkSession._set_default_and_active_session(session)
            return session

{code}


we should respect the options by invoking {{session.conf.set}} after creation.
Bug: Spark structured streaming performance regression in latency times reading/writing to kafka since 3.0.2
Status: Open | Priority: Major | Resolution: 
Created: 23/Aug/23 19:49
Affects Version/s: 2.4.8

 During a migration from spark 2.4.4 to spark 3.4.0 I have noticed slower latency times in spark structured streaming when reading and writing to kafka. I have tested using both CONTINUOUS and MICROBATCH.

In simple read and write to kafka using CONTINUOUS mode in spark 2.4.4 I usually see latency times of ~5ms in our appllication. When moving to spark 3.4.0 this increased to ~15ms.

I stripped it back to a very simple test where I send 2 data fields in csv format to a kafka topic using a simple producer. Then I have a simple consumer which reads from the input topic and writes to an output topic. The 2 fields are an ID and an amount value. I read from both topics and retrieve the kafka timestamp value for all rows. I then subtract the input timestamp from the output timestamp to get the latency. To keep things as simple as possible I am using 1 kafka partition and I am using local[1] as the spark master.

Version    latency (ms)    Trigger
2.4.4    3.25    CONTINUOUS
3.4.0    7.23    CONTINUOUS
2.4.4    640    MICROBATCH
3.4.0    693    MICROBATCH
I have tried all versions of spark 3.x and I believe this issue was introduced in 3.0.2. I also tried different versions of spark 2.4.x and I see the same behaviour when going from 2.4.7 to 2.4.8.

In the simple test I only use a few jars. One of these is spark-sql-kafka-0-10_2.12 When running on spark 3.0.2 using the 3.0.2 version of this jar I see the slower times. When I run again on spark 3.0.2 and use the 3.0.1 version of this jar I see the faster times.

The same thing happens between 2.4.7 version and the 2.4.8 version. The 2.4.8 version has the slower times.

Has anyone else observed a slow down in latency in structured streaming when reading from kafka ?

Are there any settings I need to change when moving to these versions ?
Bug: Cryptic error message when UDF associated class is not found
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/23 13:54
Affects Version/s: 3.5.0

When a Dataset API is used that either requires or is modeled as a UDF, the class defining the UDF/function should be uploaded to the service fist using the `addArtifact()` API.

When this is not done, an error is thrown. However, this error message is cryptic and is not clear about the problem.

Improve this error message to make it clear that an expected class was not found.
Bug: Continuous Structured Streaming not reporting streaming metrics
Status: Open | Priority: Major | Resolution: 
Created: 23/Aug/23 16:22
Affects Version/s: 3.4.1

Hello, we've been running spark continuous structured streaming on standalone cluster and happy with the performance however we noticed streaming metrics like input rate and process rate are not updated by the `ProgressReporter` in `ContinuousExecution` because the [`finishTrigger`|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L293] function is never invoked in `ContinuousExecution`. I'm wondering why and how may get metrics like in micro-batch structured streaming.

 

!https://preview.redd.it/mzh4oc0cbojb1.png?width=1901&format=png&auto=webp&s=8d649ae515e6adb7d6ce853802e0a4134c9fa277!!https://preview.redd.it/8ou3uyuofojb1.png?width=1523&format=png&auto=webp&s=ac6bf7fa05cb90b09cb10b9ac815f64b5e97175e!
Bug: Skip starting torch distributor log streaming server when it is not available
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/23 09:51
Affects Version/s: 0.5.0

Skip starting torch distributor log streaming server when it is not available.

 

In some cases, e.g., in a databricks connect cluster, there is some network limitation that casues starting log streaming server failure, but, this does not need to break torch distributor training routine.
Bug: Use await() instead of awaitUninterruptibly() in TransportClientFactory.createClient() 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/23 00:40
Affects Version/s: 3.3.3

This is a follow up for SPARK-44241:

That call added an `awaitUninterruptibly()` call, which I think should be a plain `await()` instead. This will prevent issues when cancelling tasks with hanging network connections. 

This issue is similar to SPARK-19529
Bug: NullPointerException is thrown when column with ROWID type contains NULL values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/23 19:47
Affects Version/s: 3.4.1

A row ID may be NULL in an Oracle table. When this is the case, the following exception is thrown:

{{[info] Cause: java.lang.NullPointerException:}}
{{{}[info] at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeGetter$12(JdbcUtils.scala:452){}}}{{{}[info] at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeGetter$12$adapted(JdbcUtils.scala:451){}}}
{{{}[info] at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:361){}}}{{{}[info] at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:343){}}}
Bug: The JIRA Python misses our assignee when it searches user again
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/23 08:14
Affects Version/s: 4.0.0

{code:java}
>>> assignee = asf_jira.user("yao")
>>> "SPARK-44801"'SPARK-44801'

>>> asf_jira.assign_issue(issue.key, assignee.name)

response text = {"errorMessages":[],"errors":{"assignee":"User 'airhot' cannot be assigned issues."}} {code}
Bug: Convert datetime to string by `to_char`/`to_varchar`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/23 11:42
Affects Version/s: 4.0.0

Expand {{to_char()}} to take date and timestamp expression as the first argument. For date and timestamp expression, the function will be equivalent to {{date_format(expr, fmt)}}

The goal is to achieve feature parity with other systems, and make the migration to Spark SQL easier:
- Snowflake: https://docs.snowflake.com/en/sql-reference/functions/to_char
- PostgreSQL: https://www.postgresql.org/docs/current/functions-formatting.html
- MariaDB: https://mariadb.com/kb/en/to_char/
- DB2: https://www.ibm.com/docs/en/db2/11.5?topic=sf-char
- Exasol: https://docs.exasol.com/db/latest/sql_references/functions/alphabeticallistfunctions/to_char%20(datetime).htm
Bug: Alter nested view fails because of HMS client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/23 16:30
Affects Version/s: 3.1.0

Currently, 
{code:java}
CREATE OR REPLACE VIEW t AS SELECT " +
        "struct(id AS `$col2`, struct(id AS `$col`) AS s1) AS s2 FROM RANGE(5)
ALTER VIEW t SET TBLPROPERTIES ('x' = 'y'){code}
would fail when calling HMS's updateTable, because HMS does not support nested struct in view. We can fix this by passing HMS an empty schema, since we store the actual view schema in the table's properties already. This fix is similar to https://github.com/apache/spark/pull/37364
Bug: Many PySpark-related test cases failed in Java 17 daily tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Aug/23 03:56
Affects Version/s: 4.0.0

The failures started two days ago.
 * [https://github.com/apache/spark/actions/runs/5837423492]
 * [https://github.com/apache/spark/actions/runs/5843658110]
 * [https://github.com/apache/spark/actions/runs/5849761680]

{code:java}
======================================================================
2381FAIL [0.008s]: test_udtf_with_analyze_taking_wrong_number_of_arguments (pyspark.sql.tests.connect.test_parity_udtf.UDTFParityTests)
2382----------------------------------------------------------------------
2383pyspark.errors.exceptions.connect.AnalysisException: [TABLE_VALUED_FUNCTION_FAILED_TO_ANALYZE_IN_PYTHON] Failed to analyze the Python user defined table function: Traceback (most recent call last):
2384  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/worker/analyze_udtf.py", line 112, in main
2385    result = handler.analyze(*args)  # type: ignore[attr-defined]
2386TypeError: analyze() takes 0 positional arguments but 1 was given
2387
2388
2389During handling of the above exception, another exception occurred:
2390
2391Traceback (most recent call last):
2392  File "/__w/spark/spark/python/pyspark/sql/tests/test_udtf.py", line 1541, in test_udtf_with_analyze_taking_wrong_number_of_arguments
2393    func(lit(1)).collect()
2394AssertionError: "analyze\(\) missing 1 required positional argument: 'b'" does not match "[TABLE_VALUED_FUNCTION_FAILED_TO_ANALYZE_IN_PYTHON] Failed to analyze the Python user defined table function: Traceback (most recent call last):
2395  File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/worker/analyze_udtf.py", line 112, in main
2396    result = handler.analyze(*args)  # type: ignore[attr-defined]
2397TypeError: analyze() takes 0 positional arguments but 1 was given
2398"
2399
2400----------------------------------------------------------------------
2401Ran 174 tests in 84.628s
2402
2403FAILED (failures=21, errors=7, skipped=6)
2404 {code}