av9ash/GitBugs huggingface dataset:
Bug: Introduce state metadata source to query the streaming state metadata information 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Nov/23 05:15
Affects Version/s: 3.5.0

Introduce a new data source so that user can query the metadata of each state store of a streaming query, the schema of the result will be

operatorId INT | operatorName STRING | stateStoreName STRING | numPartitions INT | numColsPrefixKey INT | minBatchId  LONG | minBatchId LONG

To use this source, specify the source format and checkpoint path

_df = spark.read.format(“state-metadata”).load(“/checkpointPath”)_
Bug: Support columnar/vectorized evaluation engine JDK17/21
Status: Open | Priority: Major | Resolution: 
Created: 07/Nov/23 08:06
Affects Version/s: 4.0.0

Trino uses JDK columnar/vectorized evaluation engine to improve performance:

[https://github.com/trinodb/trino/pull/19302]

[https://github.com/trinodb/trino/issues/14237]
Bug: Drop table skiped when CatalogV2Util loadTable meet unexpected Exception
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 03/Nov/23 03:34
Affects Version/s: 3.1.3

Currently  CatalogV2Util.loadTable method catch only NoSuch*Exception like below
{code:java}
  def loadTable(catalog: CatalogPlugin, ident: Identifier): Option[Table] =
    try {
      Option(catalog.asTableCatalog.loadTable(ident))
    } catch {
      case _: NoSuchTableException => None
      case _: NoSuchDatabaseException => None
      case _: NoSuchNamespaceException => None
    } {code}
It will skip drop table when conmunicate with meta time out or other Exception, because the method always return None, maybe we should catch it like below
{code:java}
def loadTable(catalog: CatalogPlugin, ident: Identifier): Option[Table] =
  try {
    Option(catalog.asTableCatalog.loadTable(ident))
  } catch {
    case e: NoSuchTableException =>  return None
    case e: NoSuchDatabaseException =>  return None
    case e: NoSuchNamespaceException =>  return None
    case e: Throwable =>  throw e
  } {code}
Bug: GROUP BY ALL don't work in ansi mode
Status: Open | Priority: Major | Resolution: 
Created: 06/Nov/23 13:26
Affects Version/s: 3.4.0

I noticed that we treat 'ALL' in 'GROUP BY ALL' as an Identifier in parsing, and this will cause problem when I turn on ansi keyword behavior
*set spark.sql.ansi.enabled=true;*
*set spark.sql.ansi.enforceReservedKeywords=true;*

spark-sql (default)> select a,b,c, count(*) from values(1,2,3)t(a,b,c) group by all;

[PARSE_SYNTAX_ERROR] Syntax error at or near 'all'.(line 1, pos 59)

== SQL ==
select a,b,c, count(*) from values(1,2,3)t(a,b,c) group by all
-----------------------------------------------------------^^^

can we allow this reserved keyword in ansi mode ?
Bug: Improve the built-in compression codecs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Nov/23 02:16
Affects Version/s: 4.0.0

Currently, Spark supported many built-in compression codecs used for I/O and storage.
There are a lot of magic strings copy from built-in compression codecs. This issue lead to developers need to manually maintain its consistency. It is easy to make mistakes and reduce development efficiency.
Bug: Introduce a mapper for hadoop compression codecs
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 01/Nov/23 12:54
Affects Version/s: 4.0.0

Currently, Spark supported partial Hadoop compression codecs, but the Hadoop supported compression codecs and spark supported are not completely one-on-one due to Spark introduce two fake compression codecs none and uncompress.

There are a lot of magic strings copy from Hadoop compression codecs. This issue lead to developers need to manually maintain its consistency. It is easy to make mistakes and reduce development efficiency.
Bug: Get SHS URL from Spark applications
Status: Open | Priority: Major | Resolution: 
Created: 03/Nov/23 07:53
Affects Version/s: 4.0.0

This is created based on the following review comments

[https://github.com/apache/spark/pull/43643#discussion_r1381212592]

> How about we get this address from the user's app first, and if not, then from the standalone master's config?

 

 

> We can add it to the [ApplicationDescription|https://github.com/apache/spark/blob/b9d379a6b84b67b29ccb578938764b888d64f293/core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala#L24-L36] ahead when you meet such a use case in the future:)
Bug: Make faulthandler a runtime configuration for Python execution in SQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Nov/23 07:15
Affects Version/s: 4.0.0

faulthanlder feature within PySpark is really useful especially to debug an errors that regular Python interpreter cannot catch out of the box such as a segmentation fault errors, see also https://github.com/apache/spark/pull/43600. It would be very useful to convert this as a runtime configuration.
Bug: Add additional test coverage for input_file_name() expr + Python UDFs
Status: Open | Priority: Major | Resolution: 
Created: 02/Nov/23 18:02
Affects Version/s: 4.0.0

https://issues.apache.org/jira/browse/SPARK-44705 introduced changes to the evaluation of the `input_file_name()` expression in the presence of Python UDFs. This was done to maintain the behavior of the `input_file_name()` expression when the execution model of the PythonRunner was made single-threaded by https://issues.apache.org/jira/browse/SPARK-44705. We should add additional test coverage for `input_file_name()` + Python UDFs to prevent future breakages.
Bug: Improve error messages when loading multiple paths in PySpark
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 01/Nov/23 23:40
Affects Version/s: 4.0.0

Currently, the error message is super confusing when a user tries to load multiple paths incorrectly.

For example, `spark.read.format("json").load("p1", "p2")` will have this error:

An error occurred while calling o36.load.
: org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: p2. Please find packages at `[https://spark.apache.org/third-party-projects.html]`. SQLSTATE: 42K02

This can be confusing but it's valid error message, as "p2" will be considered as the `format` field of the load() method.
Bug: Push down limit through Dataset.isEmpty()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Nov/23 05:51
Affects Version/s: 4.0.0

Push down LocalLimit can not optimize the case of distinct.

{code:scala}
  def isEmpty: Boolean = withAction("isEmpty",
    withTypedPlan { LocalLimit(Literal(1), select().logicalPlan) }.queryExecution) { plan =>
    plan.executeTake(1).isEmpty
  }
{code}
Bug: SPIP: ShuffleManager short name registration via SparkPlugin
Status: Open | Priority: Minor | Resolution: 
Created: 04/Nov/23 21:33
Affects Version/s: 4.0.0

We would like to make it possible for a SparkPlugin to expose custom ShuffleManager implementations using short names, in addition to default configurations for each short name in order to improve ease of use. 

Today, users leveraging a spark plugin need to set a class name under spark.plugins and if the plugin also provides a ShuffleManager, the user needs to set spark.shuffle.manager to a fully qualified class name. Additionally, users need to make sure they have set Spark configurations that are required or recommended for the shuffle manager instance. This can be cumbersome for the user, adding barriers to the pluggable interface that Spark provides.

Spark provides a short name for SortShuffleManager (“sort”) today. The idea in this SPIP is to make this set of short names extensible by a SparkPlugin.

SPIP: https://docs.google.com/document/d/1flijDjMMAAGh2C2k-vg1u651RItaRquLGB_sVudxf6I/edit?usp=sharing
Bug: Error in postgresql when pushing down filter by timestamp_ntz field
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Nov/23 21:08
Affects Version/s: 3.5.0

code to reproduce:

SparkSession sparkSession = SparkSession
.builder()
.appName("test-app")
.master("local[*]")
.config("spark.sql.timestampType", "TIMESTAMP_NTZ")
.getOrCreate();

String url = "...";

String catalogPropPrefix = "spark.sql.catalog.myc";
sparkSession.conf().set(catalogPropPrefix, JDBCTableCatalog.class.getName());
sparkSession.conf().set(catalogPropPrefix + ".url", url);

Map<String, String> options = new HashMap<>();
options.put("driver", "org.postgresql.Driver");
// options.put("pushDownPredicate", "false");  it works fine if  this line is uncommented

Dataset<Row> dataset = sparkSession.read()
.options(options)
.table("myc.demo.`My table`");

dataset.createOrReplaceTempView("view1");
String sql = "select * from view1 where `my date` = '2021-04-01 00:00:00'";
Dataset<Row> result = sparkSession.sql(sql);
result.show();
result.printSchema();

Field `my date` is of type timestamp. This code results in org.postgresql.util.PSQLException  syntax error

 

 

String sql = "select * from view1 where `my date` = to_timestamp('2021-04-01 00:00:00', 'yyyy-MM-dd HH:mm:ss')";  // this query also doesn't work

String sql = "select * from view1 where `my date` = date_trunc('DAY', to_timestamp('2021-04-01 00:00:00', 'yyyy-MM-dd HH:mm:ss'))";  // but this is OK

 

Is it a bug or I got something wrong?
Bug: Remove uneccessary toSeq when encoding Set to catalyst
Status: Open | Priority: Minor | Resolution: 
Created: 09/Nov/23 08:33
Affects Version/s: 4.0.0

Currently when encoding Sets to catalyst we first convert them into a Seq. There is no good reason to do this as the interface we are targeting for encoding is only `Iterable` which is implemented by Set. So by using Iterable instead of Seq in some places we should be able to avoid this extra copy when encoding Sets.
Bug: write empty parquet file while using partitioned write
Status: Open | Priority: Minor | Resolution: 
Created: 13/Nov/23 10:32
Affects Version/s: 3.5.0

Hi,

I'm currently using pyspark and if I try to write an empty dataframe in parquet file in a partitioned way no file is written in the target folder

df.write.mode("overwrite").partitionBy("BUSINESS_DATE").parquet("/data_dir/"+stg+"/ISS/exchange/WORK_ISSR_EOD_EXT_SETTLEMENT_CA_"+se)

 

this creates a problem because I have another job which reads the file and can't infer the schema and raises an error. I made a workaround in this way:

#implemented to manage empty data also
def write_partitioned_df(df,partition_col,partition_val,save_path):
    df.write.mode("overwrite").partitionBy(partition_col).parquet(save_path)
    if df.isEmpty():
        df=df.drop(partition_col)
        df.write.mode("overwrite").parquet(save_path+"/"+partition_col+"="+partition_val)

 

in order to write an empty parquet in the target folder but It would be great to have an option in the write function to avoid this custom implementation. I see other users interested in this feature asking on StackOverflow.

 

Thank you very much

Paride
Bug: Add Dataframe API df.explainString()
Status: Open | Priority: Minor | Resolution: 
Created: 03/Nov/23 19:25
Affects Version/s: 4.0.0

This frequently needed feature for performance optimization purposes. Users often want to look into this output in running systems and so would like to save/extract this output from running systems for later analysis.

Current API only provided for Scala i.e. 

{{df.queryExecution.toString()}}

and also not located in intuitive place where average Spark user (i.e. non Expert/Scala dev) can see immediately.

It will also avoid users using workarounds and capturing outputs with
{code:java}
with io.StringIO() as buf ...`:
    df.explain(True)
{code}
 

So, it would help users a lot have this output avalilable as:

df.explainString()

i.e. next to

df.explain()

so users can easily locate it and use.
Bug: Incorrect path for spark-hero-thin-light.jpg in spark3.5.0 website
Status: Open | Priority: Minor | Resolution: 
Created: 30/Nov/23 09:36
Affects Version/s: 3.5.0

When I visit [https://spark.apache.org/docs/3.5.0/,] spark-hero-thin-light.jpg is not found caused by [https://github.com/apache/spark-website/blob/17c63886085b582a1317a929114659f9e88822aa/site/docs/3.5.0/css/custom.css#L99,] the path should be ../images/spark-hero-thin-light.jpg
Bug: Move MockDF under PlanOnlyTestFixture
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Nov/23 00:43
Affects Version/s: 4.0.0


When you don't have the dependencies for Spark Connect, and you run the tests:

{code}
 ./python/run-tests --python-executables=python3  --testnames 'pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby_rolling'
{code}

it shows the error as below:

{code}
  File "/.../spark/python/pyspark/pandas/tests/connect/test_parity_ops_on_diff_frames_groupby_rolling.py", line 22, in <module>
    from pyspark.testing.connectutils import ReusedConnectTestCase
  File "/.../spark/python/pyspark/testing/connectutils.py", line 92, in <module>
    class PlanOnlyTestFixture(unittest.TestCase, PySparkErrorTestUtils):
  File "/.../spark/python/pyspark/testing/connectutils.py", line 93, in PlanOnlyTestFixture
    class MockDF(DataFrame):
                 ^^^^^^^^^
NameError: name 'DataFrame' is not defined
{code}

which isn't actionable.
Bug: Remove comments about JVM module options from the test submission options of `HiveExternalCatalogVersionsSuite`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Nov/23 15:42
Affects Version/s: 4.0.0

Complete TODO:
{code:java}
val args = Seq(
  "--name", "prepare testing tables",
  "--master", "local[2]",
  "--conf", s"${UI_ENABLED.key}=false",
  "--conf", s"${MASTER_REST_SERVER_ENABLED.key}=false",
  "--conf", s"${HiveUtils.HIVE_METASTORE_VERSION.key}=$hiveMetastoreVersion",
  "--conf", s"${HiveUtils.HIVE_METASTORE_JARS.key}=maven",
  "--conf", s"${WAREHOUSE_PATH.key}=${wareHousePath.getCanonicalPath}",
  "--conf", s"spark.sql.test.version.index=$index",
  "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath} " +
    // TODO SPARK-37159 Consider to remove the following
    // JVM module options once the Spark 3.2 line is EOL.
    JavaModuleOptions.defaultModuleOptions(),
  tempPyFile.getCanonicalPath) {code}
Bug: Fix the doctest in pyspark.pandas.series.Series.to_dict (Python 3.12)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Nov/23 00:48
Affects Version/s: 4.0.0

{code}
File "/__w/spark/spark/python/pyspark/pandas/series.py", line 1633, in pyspark.pandas.series.Series.to_dict
Failed example:
    s.to_dict(OrderedDict)
Expected:
    OrderedDict([(0, 1), (1, 2), (2, 3), (3, 4)])
Got:
    OrderedDict({0: 1, 1: 2, 2: 3, 3: 4})
{code}
Bug: Fix the doctest in pyspark.pandas.frame.DataFrame.to_dict (Python 3.12)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Nov/23 00:15
Affects Version/s: 4.0.0

{code}
File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 2515, in pyspark.pandas.frame.DataFrame.to_dict
Failed example:
    df.to_dict(into=OrderedDict)
Expected:
    OrderedDict([('col1', OrderedDict([('row1', 1), ('row2', 2)])), ('col2', OrderedDict([('row1', 0.5), ('row2', 0.75)]))])
Got:
    OrderedDict({'col1': OrderedDict({'row1': 1, 'row2': 2}), 'col2': OrderedDict({'row1': 0.5, 'row2': 0.75})})
{code}
Bug: Replace `df.take(1).toSeq(0)` with `df.first()` in `ProtobufFunctionsSuite`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Nov/23 07:06
Affects Version/s: 4.0.0

In `ProtobufFunctionsSuite`, there are some cases where `df.take(1).toSeq(0)` is used to get the first row in the DataFrame. This can be achieved by using the `.first()` API, which looks clearer and more concise.
Bug: Reduce stack depth by replace (string|array).size with (string|array).length
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Nov/23 09:58
Affects Version/s: 4.0.0

There are a lot of (string|array).size called.
In fact, the size calls the underlying length, this behavior increase the stack length.
We should call (string|array).length directly.

We also get the compile waring Replace .size with .length on arrays and strings
Bug: Reeanble `pyspark.tests.test_worker.WorkerSegfaultNonDaemonTest.test_python_segfault` with Python 3.12
Status: Open | Priority: Minor | Resolution: 
Created: 28/Nov/23 02:32
Affects Version/s: 4.0.0

Reeanble https://issues.apache.org/jira/browse/SPARK-46127.
Bug: Use the Separators API instead of the String API to construct the DefaultPrettyPrinter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Nov/23 06:58
Affects Version/s: 4.0.0

{code:java}
/**
 * Constructor that specifies separator String to use between root values;
 * if null, no separator is printed.
 *<p>
 * Note: simply constructs a {@link SerializedString} out of parameter,
 * calls {@link #DefaultPrettyPrinter(SerializableString)}
 *
 * @param rootSeparator String to use as root value separator
 * @deprecated in 2.16. Use the Separators API instead.
 */
@Deprecated
public DefaultPrettyPrinter(String rootSeparator) {
    this((rootSeparator == null) ? null : new SerializedString(rootSeparator));
} {code}
Bug: append to parquet file with column type changed corrupts fie
Status: Open | Priority: Minor | Resolution: 
Created: 24/Nov/23 15:06
Affects Version/s: 3.3.0

from pyspark.sql.functions import *
from pyspark.sql.types import *

fnBad = "dbfs:/tmp/richard.gooding@os.uk/test_bad_parquet/f1"
df = spark.createDataFrame( [ ["aaaa" ] ] ).select( col("_1").alias("aa") )
df.printSchema()

fmt = "parquet"
# fmt = "delta"
df.write.mode("overwrite").format( fmt ) .save( fnBad )
df.show()

df = df.withColumn( "aa", struct( col("aa")) ) # change type of column - error on load
df.printSchema()
df.show()
df.write.mode("append").format( fmt).save( fnBad ) # format = delta :   "AnalysisException: Failed to merge fields 'aa' and 'aa'. Failed to merge incompatible data types StringType and StructType(StructField(aa,StringType,true))"
# df.write.mode("append").option("mergeSchema", "true").format(fmt).save( fnBad ) # gives a different error, but only when dataframe read

print(" --- at df 2 --- ")
df2 = spark.read.format(fmt).load( fnBad )
# df2 = spark.read.option("mergeSchema", "true").format(fmt).load( fnBad )
df2.show()  # this will error -
Bug: Fix protobuf string representation for Pandas Functions API with Spark Connect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/23 01:07
Affects Version/s: 4.0.0

{code}
df = spark.range(1)
df.mapInPandas(lambda x: x, df.schema)._plan.print()
{code}

prints as below. It should includes functions.

{code}
<MapPartitions is_barrier='False'>
  <Range start='0', end='1', step='1', num_partitions='None'>
{code}
Bug: Improve error messages related to argument types in cute, rollup, groupby, and pivot
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Nov/23 04:18
Affects Version/s: 4.0.0

{code}
>>> spark.range(1).cube(cols=1.2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../python/pyspark/sql/connect/dataframe.py", line 544, in cube
    raise PySparkTypeError(
pyspark.errors.exceptions.base.PySparkTypeError: [NOT_COLUMN_OR_STR] Argument `cube` should be a Column or str, got float.
{code}

{code}
>>> help(spark.range(1).cube)
Help on method cube in module pyspark.sql.connect.dataframe:

cube(*cols: 'ColumnOrName') -> 'GroupedData' method of pyspark.sql.connect.dataframe.DataFrame instance
    Create a multi-dimensional cube for the current :class:`DataFrame` using
    the specified columns, allowing aggregations to be performed on them.

    .. versionadded:: 1.4.0

    .. versionchanged:: 3.4.0
{code}

it has to be {cols}
Bug: EPOCH extract field is not supported
Status: Open | Priority: Minor | Resolution: 
Created: 17/Nov/23 16:50
Affects Version/s: 3.5.0

{code:java}
spark.sql("""SELECT EXTRACT(EPOCH FROM "2017-06-15")""").show()
org.apache.spark.sql.AnalysisException: [INVALID_EXTRACT_FIELD] Cannot extract `EPOCH` from "2017-06-15".; line 1 pos 7 {code}
Bug: Remove Java version check from `IsolatedClientLoader`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Nov/23 06:41
Affects Version/s: 4.0.0

{code:java}
val rootClassLoader: ClassLoader =
  if (SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9)) {
    // In Java 9, the boot classloader can see few JDK classes. The intended parent
    // classloader for delegation is now the platform classloader.
    // See http://java9.wtf/class-loading/
    val platformCL =
    classOf[ClassLoader].getMethod("getPlatformClassLoader").
      invoke(null).asInstanceOf[ClassLoader]
    // Check to make sure that the root classloader does not know about Hive.
    assert(Try(platformCL.loadClass("org.apache.hadoop.hive.conf.HiveConf")).isFailure)
    platformCL
  } else {
    // The boot classloader is represented by null (the instance itself isn't accessible)
    // and before Java 9 can see all JDK classes
    null
  } {code}
Spark 4.0.0 has a minimum requirement of Java 17, so the version check for Java 9 is not necessary.
Bug: Redundant parameter passing
Status: Open | Priority: Minor | Resolution: 
Created: 08/Nov/23 07:19
Affects Version/s: 3.5.0



org.apache.spark.ui.jobs.AllJobsPage.scala

{code:scala}
// UIUtils.makeDescription 
//plainText is a redundant parameter and is set to false by default in method signatures

private def jobRow(jobData: v1.JobData): JobTableRowData = {
xxx
val jobDescription =
      UIUtils.makeDescription(
        jobData.description.getOrElse(lastStageDescription),
        basePath,
        plainText = false)
xx
}
{code}
Bug: The isFile method in the FileSystem is Deprecated
Status: Open | Priority: Minor | Resolution: 
Created: 07/Nov/23 08:06
Affects Version/s: 3.5.0

* The isFile method in the FileSystem is Deprecated,Use {@link #getFileStatus(Path)} instead;
HdfsUtils.scala
{code:scala}
// Some comments here
def getOutputStream(path: String, conf: Configuration): FSDataOutputStream = {
    val dfsPath = new Path(path)
    val dfs = getFileSystemForPath(dfsPath, conf)
    // If the file exists and we have append support, append instead of creating a new file
    val stream: FSDataOutputStream = {
      if (dfs.isFile(dfsPath)) {
.....
{code}
Bug: Introduce two helper methods for `QueryTest` that accept the `Array` type `expectedAnswer`
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 06/Nov/23 00:55
Affects Version/s: 4.0.0

They are used to bridge
{code:java}
protected def checkAnswer(df: => DataFrame, expectedAnswer: Seq[Row]): Unit {code}
and
{code:java}
def checkAnswer(df: DataFrame, expectedAnswer: Seq[Row], checkToRDD: Boolean = true): Unit {code}
to reduce compilation warnings related to `method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated`
Bug: Remove the defensive null check added in SPARK-39553.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Nov/23 03:45
Affects Version/s: 4.0.0

{code:java}
def unregisterShuffle(shuffleId: Int): Unit = {
    shuffleStatuses.remove(shuffleId).foreach { shuffleStatus =>
      // SPARK-39553: Add protection for Scala 2.13 due to https://github.com/scala/bug/issues/12613
      // We should revert this if Scala 2.13 solves this issue.
      if (shuffleStatus != null) {
        shuffleStatus.invalidateSerializedMapOutputStatusCache()
        shuffleStatus.invalidateSerializedMergeOutputStatusCache()
      }
    }
  } {code}
This issue has been fixed in Scala 2.13.9.