av9ash/GitBugs huggingface dataset:
Bug: SortMergeJoin should respect the streamed side ordering
Status: Open | Priority: Major | Resolution: 
Created: 14/Aug/23 14:41
Affects Version/s: 3.5.0

In each partition, SortMergeJoin will compute one by one from the streamed side, so we could respect the streamed side ordering to remove unnecessary sort.
For example, when REQUIRE_ALL_CLUSTER_KEYS_FOR_CO_PARTITION is false:


{code:java}
SELECT *
FROM (
    SELECT t1.*, row_number() over(partition by a order by b) rn
    FROM values(1, 1) t1(a, b)
) t1
JOIN values(1) t2(c)
ON a = c
JOIN values(1, 1) t3(d, e)
ON a = d
AND b = e
{code}


Plan:

{code:java}
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [a#220, b#221], [d#223, e#224], Inner
   :- Sort [a#220 ASC NULLS FIRST, b#221 ASC NULLS FIRST], false, 0
   :  +- SortMergeJoin [a#220], [c#222], Inner
   :     :- Window [row_number() windowspecdefinition(a#220, b#221 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS rn#218], [a#220], [b#221 ASC NULLS FIRST]
   :     :  +- Sort [a#220 ASC NULLS FIRST, b#221 ASC NULLS FIRST], false, 0
   :     :     +- Exchange hashpartitioning(a#220, 5), ENSURE_REQUIREMENTS, [plan_id=93]
   :     :        +- LocalTableScan [a#220, b#221]
   :     +- Sort [c#222 ASC NULLS FIRST], false, 0
   :        +- Exchange hashpartitioning(c#222, 5), ENSURE_REQUIREMENTS, [plan_id=98]
   :           +- LocalTableScan [c#222]
   +- Sort [d#223 ASC NULLS FIRST, e#224 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(d#223, 5), ENSURE_REQUIREMENTS, [plan_id=104]
         +- LocalTableScan [d#223, e#224]
{code}


The second *Sort [a#220 ASC NULLS FIRST, b#221 ASC NULLS FIRST], false, 0* in the plan could be removed
Bug: Encoders.bean does not support superclasses with generic type arguments
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/23 11:54
Affects Version/s: 3.4.1

As per SPARK-44634 another unsupported feature of bean encoder is when the superclass of the bean has generic type arguments. For example:
{code:java}
class JavaBeanWithGenericsA<T> {
    public T getPropertyA() {
        return null;
    }

    public void setPropertyA(T a) {

    }
}

class JavaBeanWithGenericBase extends JavaBeanWithGenericsA<String> {
}

Encoders.bean(JavaBeanWithGenericBase.class); // Exception

{code}
Bug: Encoders.bean does no longer support nested beans with type arguments
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/23 09:13
Affects Version/s: 3.4.1

Hi,

  while upgrading a project from spark 2.4.0 to 3.4.1 version, I have encountered the same problem described in [java - Encoders.bean attempts to check the validity of a return type considering its generic type and not its concrete class, with Spark 3.4.0 - Stack Overflow|https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge].

Put it short, starting from Spark 3.4.x Encoders.bean throws an exception when the passed class contains a field whose type is a nested bean with type arguments:

 
{code:java}
class A<T> {
   T value;
   // value getter and setter
}

class B {
   A<String> stringHolder;
   // stringHolder getter and setter
}

Encoders.bean(B.class); // throws "SparkUnsupportedOperationException: [ENCODER_NOT_FOUND]..."{code}
 

 

It looks like this is a regression introduced with [SPARK-42093 SQL Move JavaTypeInference to AgnosticEncoders|https://github.com/apache/spark/commit/18672003513d5a4aa610b6b94dbbc15c33185d3#diff-1191737b908340a2f4c22b71b1c40ebaa0da9d8b40c958089c346a3bda26943b] while getting rid of TypeToken, that somehow managed that case.
Bug: Reducing the CI time for slow pyspark-pandas-connect tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/23 09:17
Affects Version/s: 4.0.0

pyspark-pandas-connect test takes more than 3 hours in Github Actions, so we might need to reduce the execution time. See https://github.com/apache/spark/actions/runs/5989124806/job/16245001034
Bug: DiskBlockManager should check and be able to handle stale directories
Status: Open | Priority: Major | Resolution: 
Created: 02/Aug/23 04:14
Affects Version/s: 3.4.1

The subDir in the memory cache could be stale, for example, after a damaged disk repair or replacement. This dir could be accessed subsequently by others. Especially,  `filename` generated by `RDDBlockId` is unchanged between task reties, so it probably attempts to access the same subDir repeatedly. Therefore, it is necessary to check if the subDir exists. If it is stale and the hardware has been recovered without data and directories, we will recreate the subDir to prevent FileNotFoundException during writing.
Bug: Cached DataFrame keeps growing
Status: Open | Priority: Major | Resolution: 
Created: 21/Aug/23 23:26
Affects Version/s: 3.3.0

Scenario :

We have a kafka streaming application where the data lookups are happening by joining  another DF which is cached, and the caching strategy is MEMORY_AND_DISK.

However the size of the cached DataFrame keeps on growing for every micro batch the streaming application process and that's being visible under storage tab.

A similar stack overflow thread was already raised.

https://stackoverflow.com/questions/55601779/spark-dataframe-cache-keeps-growing
Bug: CONV produces incorrect result near Long.MIN_VALUE, fails to detect overflow
Status: Open | Priority: Major | Resolution: 
Created: 24/Aug/23 07:49
Affects Version/s: 3.4.1

Signed conversion does not detect overflow 
{code:java}
>>> spark.conf.set('spark.sql.ansi.enabled', True)
>>> sql("SELECT conv('-9223372036854775809', 10, -10)").show(truncate=False)
+-----------------------------------+
|conv(-9223372036854775809, 10, -10)|
+-----------------------------------+
|-9223372036854775807               |
+-----------------------------------+
{code}

Unsigned conversion produces -1 but does not throw in the ANSI mode
{code}
>>> sql("SELECT conv('-9223372036854775809', 10, 10)").show(truncate=False)
+----------------------------------+
|conv(-9223372036854775809, 10, 10)|
+----------------------------------+
|18446744073709551615              |
+----------------------------------+
{code}
Bug: Code-gen CodegenFallback expression in WholeStageCodegen if possible
Status: Open | Priority: Major | Resolution: 
Created: 11/Aug/23 08:52
Affects Version/s: 3.5.0

Now both WholeStageCodegen framework and SubExpressionElimination framework does not support CodegenFallback expression, but the CodegenFallback expression which contains nullSafeEval method could gen-code just like common expressions, now they are always be executed in a new SpecificUnsafeProjection class, and we can not eliminate the sub expressions.

For example:
SQL:
{code:sql}
SELECT from_json(regexp_replace(s, 'a', 'x'), 'x INT, b DOUBLE').x,
       from_json(regexp_replace(s, 'a', 'x'), 'x INT, b DOUBLE').b
FROM values('{"a":1, "b":0.8}') t(s)
{code}
plan:
{code:java}
*(1) Project [from_json(StructField(x,IntegerType,true), regexp_replace(s#218, a, x, 1), Some(America/Los_Angeles)).x AS from_json(regexp_replace(s, a, x, 1)).x#219, from_json(StructField(b,DoubleType,true), regexp_replace(s#218, a, x, 1), Some(America/Los_Angeles)).b AS from_json(regexp_replace(s, a, x, 1)).b#220]
+- *(1) LocalTableScan [s#218]
{code}
Due to expression org.apache.spark.sql.catalyst.expressions.JsonToStructs is CodegenFallback expression, so we can not reuse the result of {*}regexp_replace(s, 'a', 'x'){*}.
We can support expression org.apache.spark.sql.catalyst.expressions.JsonToStructs code-gen in WholeStageCodegen framework, and then reuse the result of {*}regexp_replace(s, 'a', 'x'){*}.
Bug: Convert HashAggregate to SortAggregate if all grouping expressions are in child output orderings
Status: Open | Priority: Major | Resolution: 
Created: 18/Aug/23 12:06
Affects Version/s: 3.5.0

If all the grouping expressions of Aggregate could be found in the child output orderings, it still can be convert to SortAggregate.
For example:
{code:java}
SELECT a, b, count(1)
FROM values(1, 1, 1), (2, 2, 2) t1(a, b, c)
JOIN values(1, 1, 1), (2, 2, 2) t2(d, e, f)
ON a = d
AND b = e
GROUP by b, a{code}
The grouping expressions are b, a, and the child output orderings are a.asc, b.asc, it still can be convert to SortAggregate.

If all the grouping expressions of Aggregate are semanticEquals with the child output orderings, it can be replaced to SortAggregate.
For example:
{code:java}
SELECT a, b, count(1)
FROM values(1, 1, 1), (2, 2, 2) t1(a, b, c)
JOIN values(1, 1, 1), (2, 2, 2) t2(d, e, f)
ON a = d
AND b = e
GROUP by a, b, d{code}
The grouping expressions are a, b, d, and the child output orderings are a.asc, b.asc, but we still can find d in a.asc.children, so it can be convert to SortAggregate.
Bug: Plugin API for PySpark and SparkR workers
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/23 21:41
Affects Version/s: 3.4.1

An API to customize Python and R workers allows for extensibility beyond what can be expressed via static configs and environment variables like, e.g., {{spark.pyspark.python}}.

A use case for this is overriding {{PATH}} when using {{spark.archives}} with, say, conda-pack (as documented [here|https://spark.apache.org/docs/3.1.1/api/python/user_guide/python_packaging.html#using-conda]). Some packages rely on binaries. And if we want to use those packages in Spark, we need to include their binaries in the {{PATH}}.

But we can't set the {{PATH}} via some config because 1) the environment with its binaries may be at a dynamic location (archives are unpacked on the driver [into a directory with random name|https://github.com/apache/spark/blob/5db87787d5cc1cefb51ec77e49bac7afaa46d300/core/src/main/scala/org/apache/spark/SparkFiles.scala#L33-L37]), and 2) we may not want to override the {{PATH}} that's pre-configured on the hosts.

Other use cases unlocked by this include overriding the executable dynamically (e.g., to select a version) or forking/redirecting the worker's output stream.
Bug: Runtime filter should supports reuse exchange if it can reduce the data size of application side
Status: Open | Priority: Major | Resolution: 
Created: 12/Aug/23 04:43
Affects Version/s: 4.0.0

Currently, Spark runtime filter only supports using the subquery on one table.
In fact, we can reuse the exchange, even if it is a shuffle exchange.
If the shuffle exchange come from a join which has one side with selective predicates, so the results of the join can be used to prune the data amount of the application side.
Bug: Fix ArrayIndexOutOfBoundsException in conv()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/23 21:05
Affects Version/s: 3.0.3


{code:scala}
scala> sql(s"SELECT CONV('${Long.MinValue}', 10, -2)").show(false)
java.lang.ArrayIndexOutOfBoundsException: -1
  at org.apache.spark.sql.catalyst.util.NumberConverter$.convert(NumberConverter.scala:183)
  at org.apache.spark.sql.catalyst.expressions.Conv.nullSafeEval(mathExpressions.scala:463)
  at org.apache.spark.sql.catalyst.expressions.TernaryExpression.eval(Expression.scala:821)
  at org.apache.spark.sql.catalyst.expressions.ToPrettyString.eval(ToPrettyString.scala:57)
  at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.org$apache$spark$sql$catalyst$optimizer$ConstantFolding$$constantFolding(expressions.scala:81)
  at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.$anonfun$constantFolding$4(expressions.scala:91)
{code}
Bug: Add user guide for type mappings between Spark and Python data types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Aug/23 02:21
Affects Version/s: 4.0.0

Currently, the PySpark documentation does not cover the data type mapping between Spark data types and Python data types. This mapping can be useful for users of Python UDFs/UDTFs.

There's a document detailing type mapping in the Spark documentation: [https://spark.apache.org/docs/3.4.1/sql-ref-datatypes.html]

We should create a dedicated documentation page dedicated to type mappings.
Bug: Introduce CLUSTER BY SQL clause to CREATE/REPLACE TABLE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/23 02:42
Affects Version/s: 4.0.0

This proposes to introduce CLUSTER BY clause to CREATE/REPLACE SQL syntax:
{code:java}
CREATE TABLE tbl(a int, b string) CLUSTER BY (a, b){code}
This doesn't introduce a default implementation for clustering, but it's up to the catalog/datasource implementation to utilize the clustering information (e.g., Delta, Iceberg, etc.).
Bug: Make PythonRunner single-threaded
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/23 14:50
Affects Version/s: 3.5.0


PythonRunner, a utility that executes Python UDFs in Spark, uses two threads in a producer-consumer model today. This multi-threading model is problematic and confusing as Spark's execution model within a task is commonly understood to be single-threaded. 
More importantly, this departure of a double-threaded execution resulted in a series of customer issues involving [race conditions|https://issues.apache.org/jira/browse/SPARK-33277] and [deadlocks|https://issues.apache.org/jira/browse/SPARK-38677] between threads as the code was hard to reason about. There have been multiple attempts to reign in these issues, viz., [fix 1|https://issues.apache.org/jira/browse/SPARK-22535], [fix 2|https://github.com/apache/spark/pull/30177], [fix 3|https://github.com/apache/spark/commit/243c321db2f02f6b4d926114bd37a6e74c2be185]. Moreover, the fixes have made the code base somewhat abstruse by introducing multiple daemon [monitor threads|https://github.com/apache/spark/blob/a3a32912be04d3760cb34eb4b79d6d481bbec502/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala#L579] to detect deadlocks.
Bug: Add option to use Java tmp dir for RocksDB state store
Status: Open | Priority: Major | Resolution: 
Created: 02/Aug/23 17:47
Affects Version/s: 3.4.1

Currently local RocksDB state is stored in a local directory given by Utils.getLocalDir. On yarn this is a directory created inside the root application folder such as

{{/tmp/nm-local-dir/usercache/<user>/appcache/<app_id>/}}

The problem with this is that if an executor crashes for some reason (like OOM) and the shutdown hooks don't get run, this directory will stay around forever until the application finishes, which can cause jobs to slowly accumulate more and more temporary space until finally the node manager goes unhealthy.

Because this data will only ever be accessed by the executor that created this directory, it would make sense to store the data inside the container folder, which will always get cleaned up by the node manager when that yarn container gets cleaned up. Yarn sets the `java.io.tmpdir` to a path inside this directory, such as

{{/tmp/nm-local-dir/usercache/<user>/appcache/<app_id>/<container_id>/tmp/}}

I'm not sure the behavior for other resource managers, so this could be an opt-in config that can be specified.
Bug: SPIP: Python Stored Procedures
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 30/Aug/23 22:19
Affects Version/s: 4.0.0

Stored procedures are an extension of the ANSI SQL standard. They play a crucial role in improving the capabilities of SQL by encapsulating complex logic into reusable routines. 

This proposal aims to extend Spark SQL by introducing support for stored procedures, starting with Python as the procedural language. This addition will allow users to execute procedural programs, leveraging programming constructs of Python to perform tasks with complex logic. Additionally, users can persist these procedural routines in catalogs such as HMS for future reuse. By providing this functionality, we intend to seamlessly empower Spark users to integrate with Python routines within their SQL workflows.

{*}SPIP{*}: [https://docs.google.com/document/d/1ce2EZrf2BxHu7TjfGn4TgToK3TBYYzRkmsIVcfmkNzE/edit?usp=sharing]
Bug: Add documentation for type casting rules in Python UDFs/UDTFs
Status: Open | Priority: Major | Resolution: 
Created: 09/Aug/23 02:27
Affects Version/s: 4.0.0

In addition to type mappings between Spark data types and Python data types (SPARK-44733), we should add the type casting rules for regular and arrow-optimized Python UDFs/UDTFs. 

We currently have this table in code:
 * Arrow: [https://github.com/apache/spark/blob/master/python/pyspark/sql/pandas/functions.py#L311-L329]
 * Python UDF: [https://github.com/apache/spark/blob/master/python/pyspark/sql/udf.py#L101-L116]

We should add a proper documentation page for the type casting rules.
Bug: Runtime Filter supports passing equivalent creation side expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/23 03:46
Affects Version/s: 4.0.0


{code:java}
SELECT
  d_year,
  i_brand_id,
  i_class_id,
  i_category_id,
  i_manufact_id,
  cs_quantity - COALESCE(cr_return_quantity, 0) AS sales_cnt,
  cs_ext_sales_price - COALESCE(cr_return_amount, 0.0) AS sales_amt
FROM catalog_sales
  JOIN item ON i_item_sk = cs_item_sk
  JOIN date_dim ON d_date_sk = cs_sold_date_sk
  LEFT JOIN catalog_returns ON (cs_order_number = cr_order_number
    AND cs_item_sk = cr_item_sk)
WHERE i_category = 'Books'
{code}
Bug: Support ignoreCorruptFiles for multiline CSV
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/23 16:42
Affects Version/s: 3.5.0

Today, `ignoreCorruptFiles` does not work well for multiline CSV mode.
{code:java}
spark.conf.set("spark.sql.files.ignoreCorruptFiles", "true")val testCorruptDF0 = spark.read.option("ignoreCorruptFiles", "true").option("multiline", "true").csv("/tmp/sourcepath/").show() {code}
It throws an exception instead of ignoring silently:
{code:java}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4940.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4940.0 (TID 4031) (10.68.177.106 executor 0): com.univocity.parsers.common.TextParsingException: java.lang.IllegalStateException - Error reading from input
Parser Configuration: CsvParserSettings:
	Auto configuration enabled=true
	Auto-closing enabled=true
	Autodetect column delimiter=false
	Autodetect quotes=false
	Column reordering enabled=true
	Delimiters for detection=null
	Empty value=
	Escape unquoted values=false
	Header extraction enabled=null
	Headers=null
	Ignore leading whitespaces=false
	Ignore leading whitespaces in quotes=false
	Ignore trailing whitespaces=false
	Ignore trailing whitespaces in quotes=false
	Input buffer size=1048576
	Input reading on separate thread=false
	Keep escape sequences=false
	Keep quotes=false
	Length of content displayed on error=1000
	Line separator detection enabled=true
	Maximum number of characters per column=-1
	Maximum number of columns=20480
	Normalize escaped line separators=true
	Null value=
	Number of records to read=all
	Processor=none
	Restricting data in exceptions=false
	RowProcessor error handler=null
	Selected fields=none
	Skip bits as whitespace=true
	Skip empty lines=true
	Unescaped quote handling=STOP_AT_DELIMITERFormat configuration:
	CsvFormat:
		Comment character=#
		Field delimiter=,
		Line separator (normalized)=\n
		Line separator sequence=\n
		Quote character="
		Quote escape character=\
		Quote escape escape character=null
Internal state when error was thrown: line=0, column=0, record=0
	at com.univocity.parsers.common.AbstractParser.handleException(AbstractParser.java:402)
	at com.univocity.parsers.common.AbstractParser.beginParsing(AbstractParser.java:277)
	at com.univocity.parsers.common.AbstractParser.beginParsing(AbstractParser.java:843)
	at org.apache.spark.sql.catalyst.csv.UnivocityParser$$anon$1.<init>(UnivocityParser.scala:463)
	at org.apache.spark.sql.catalyst.csv.UnivocityParser$.convertStream(UnivocityParser.scala:46... {code}
It is because the multiline parsing uses a different RDD (`BinaryFileRDD`) which does not go through `FileScanRDD`. We could potentially add this support to `BinaryFileRDD`, or even reuse the `FileScanRDD` for multiline parsing mode.
Bug: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Status: Open | Priority: Major | Resolution: 
Created: 04/Aug/23 16:25
Affects Version/s: 3.2.1

We get the following error from our Pyspark application in Production env:

_java.lang.OutOfMemoryError: Requested array size exceeds VM limit_

I simplified the code we used and shared it below so you can easily investigate the issue.

We use Pyspark to read 900 MB text file which has one record. We use foreach function to iterate over the Datafreme and apply some high order function. The error occurs once foreach action is triggered. I think the issue is related to the integer data type of the bytes array used to hold the serialized dataframe. Since the dataframe record was too big, it seems the serialized record exceeded the max integer value, hence the error occurred. 

Note that the same error happens when using foreachBatch function with writeStream. 

Our prod data has many records larger than 100 MB.  Appreciate your help to provide a fix or a solution to that issue.

 

*Find below the code snippet:*
from pyspark.sql import SparkSession,functions as f
 
def check_file_name(row):
    print("check_file_name called")
 
def main():
    spark=SparkSession.builder.enableHiveSupport().getOrCreate()
inputPath = "s3://bucket-name/common/source/"
    inputDF = spark.read.text(inputPath, wholetext=True)
    inputDF = inputDF.select(f.date_format(f.current_timestamp(), 'yyyyMMddHH').astype('string').alias('insert_hr'),
                        f.col("value").alias("raw_data"),
                        f.input_file_name().alias("input_file_name"))
    inputDF.foreach(check_file_name)
 
if __name__ == "__main__":
    main()
*Find below spark-submit command used:*

spark-submit --master yarn --conf spark.serializer=org.apache.spark.serializer.KryoSerializer  --num-executors 15 --executor-cores 4 --executor-memory 20g --driver-memory 20g --name haitham_job --deploy-mode cluster big_file_process.py
Bug: Small tweaks to attaching ExecuteGrpcResponseSender to ExecuteResponseObserver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/23 17:24
Affects Version/s: 4.0.0

Small improvements can be made to the way new ExecuteGrpcResponseSender is attached to observer.
 * Since now we have addGrpcResponseSender in ExecuteHolder, it should be ExecuteHolder responsibility to interrupt the old sender and that there is only one at a time, and to ExecuteResponseObserver's responsibility
 * executeObserver is used as a lock for synchronization. An explicit lock object could be better.
Bug: Add canonical links to the PySpark docs page
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/23 23:03
Affects Version/s: 4.0.0

We should add the canonical link to the PySpark docs page [https://spark.apache.org/docs/latest/api/python/index.html] so that the search engine can return the latest PySpark docs.

Then, we need to update all released documentation pages to add the canonical link pointing to the latest spark documentation of the API (such as group by). Currently, if you Google pyspark groupby, Google will return the docs page from 3.1.1, which is not ideal.
Bug: DS V2 supports push down V2 UDF that has magic method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/23 14:58
Affects Version/s: 3.4.1

Right now we only support pushing down the V2 UDF that has not a magic method. Because the V2 UDF will be analyzed into the `ApplyFunctionExpression` which could be translated and pushed down. However, a V2 UDF that has the magic method will be analyzed into `StaticInvoke` or `Invoke` that can not be translated into V2 expression and then can not be pushed down to the data source. The magic method is suggested. So this PR adds the support of pushing down the V2 UDF that has a magic method.