av9ash/GitBugs huggingface dataset:
Bug: StringTrim & StringTrimLeft/Right/Both (binary & lowercase collation only)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/24 11:54
Affects Version/s: 4.0.0

Enable collation support for the *StringTrim* built-in string function in Spark (including {*}StringTrimBoth{*}, {*}StringTrimLeft{*}, {*}StringTrimRight{*}). First confirm what is the expected behaviour for these functions when given collated strings, and then move on to implementation and testing. One way to go about this is to consider using {_}StringSearch{_}, an efficient ICU service for string matching. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|[https://www.postgresql.org/docs/]].

 

The goal for this Jira ticket is to implement the *StringTrim* function so it supports binary & lowercase collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class, as well as _StringSearch_ using the [ICU user guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html] and [ICU docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html]. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Remove experimental tag from toArrow() PySpark DataFrame method
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 19/Mar/24 16:50
Affects Version/s: 3.5.1

As a follow-up to SPARK-47365:

What is needed to consider making the *toArrow()* PySpark DataFrame non-experimental?

What can the Apache Arrow developers do to help with this?
Bug: Add toArrow() DataFrame method to PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/24 18:59
Affects Version/s: 3.5.1

Over in the Apache Arrow community, we hear from a lot of users who want to return the contents of a PySpark DataFrame as a [PyArrow Table|https://arrow.apache.org/docs/python/generated/pyarrow.Table.html]. Currently the only documented way to do this is:

*PySpark DataFrame* --> *pandas DataFrame* --> *PyArrow Table*

This adds significant overhead compared to going direct from PySpark DataFrame to PyArrow Table. Since [PySpark already goes through PyArrow to convert to pandas|https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html], would it be possible to publicly expose a *toArrow()* method of the Spark DataFrame class?
Bug: Allow trailing comma in column definition list
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/24 08:55
Affects Version/s: 4.0.0

When experimenting with SQL queries, it's common to accidentally leave a trailing comma after the last column in a table definition, resulting in syntax errors. This often happens when developers comment out the last column and forget to remove the trailing comma, leading to unexpected syntax issues when creating tables.
Bug: SubstringIndex
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/24 13:50
Affects Version/s: 4.0.0

Enable collation support for the *SubstringIndex* built-in string function in Spark. First confirm what is the expected behaviour for these functions when given collated strings, and then move on to implementation and testing. One way to go about this is to consider using {_}StringSearch{_}, an efficient ICU service for string matching. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *SubstringIndex* functions so that they support all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class, as well as _StringSearch_ using the [ICU user guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html] and [ICU docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html]. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: StringLocate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/24 13:53
Affects Version/s: 4.0.0

Enable collation support for the *StringLocate* built-in string function in Spark. First confirm what is the expected behaviour for these functions when given collated strings, and then move on to implementation and testing. One way to go about this is to consider using {_}StringSearch{_}, an efficient ICU service for string matching. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *StringLocate* functions so that they support all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class, as well as _StringSearch_ using the [ICU user guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html] and [ICU docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html]. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: SQLServer does not support LIKE operator in binary comparison
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/24 12:42
Affects Version/s: 4.0.0

When pushing Spark query to MsSqlServer engine we sometimes construct SQL query that has a LIKE operator as a part of the binary comparison operation, which is not permitted in SQL Server syntax. 

For example a query 
{code:java}
SELECT * FROM people WHERE (name LIKE "s%") = 1{code}
will not execute on MsSQLServer.

These queries should be detected and not pushed to execution in MsSqlServer.
Bug: StringReplace (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/24 06:32
Affects Version/s: 4.0.0

Enable collation support for the *StringReplace* built-in string function in Spark. First confirm what is the expected behaviour for this function when given collated strings, and then move on to implementation and testing. One way to go about this is to consider using {_}StringSearch{_}, an efficient ICU service for string matching. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *StringReplace* function so it supports all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class, as well as _StringSearch_ using the [ICU user guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html] and [ICU docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html]. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Cache miss for queries using JOIN LATERAL with join condition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 20:21
Affects Version/s: 3.4.2

For example:
{noformat}
CREATE or REPLACE TEMP VIEW t1(c1, c2) AS VALUES (0, 1), (1, 2);
CREATE or REPLACE TEMP VIEW t2(c1, c2) AS VALUES (0, 1), (1, 2);

create or replace temp view v1 as
select *
from t1
join lateral (
  select c1 as a, c2 as b
  from t2)
on c1 = a;

cache table v1;

explain select * from v1;
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- BroadcastHashJoin [c1#180], [a#173], Inner, BuildRight, false
   :- LocalTableScan [c1#180, c2#181]
   +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=113]
      +- LocalTableScan [a#173, b#174]
{noformat}
Note that there is no {{InMemoryRelation}}.

However, if you move the join condition into the subquery, the cached plan is used:
{noformat}
CREATE or REPLACE TEMP VIEW t1(c1, c2) AS VALUES (0, 1), (1, 2);
CREATE or REPLACE TEMP VIEW t2(c1, c2) AS VALUES (0, 1), (1, 2);

create or replace temp view v2 as
select *
from t1
join lateral (
  select c1 as a, c2 as b
  from t2
  where t1.c1 = t2.c1);

cache table v2;

explain select * from v2;
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Scan In-memory table v2 [c1#176, c2#177, a#178, b#179]
      +- InMemoryRelation [c1#176, c2#177, a#178, b#179], StorageLevel(disk, memory, deserialized, 1 replicas)
            +- AdaptiveSparkPlan isFinalPlan=true
               +- == Final Plan ==
                  *(1) Project [c1#26, c2#27, a#19, b#20]
                  +- *(1) BroadcastHashJoin [c1#26], [c1#30], Inner, BuildLeft, false
                     :- BroadcastQueryStage 0
                     :  +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=37]
                     :     +- LocalTableScan [c1#26, c2#27]
                     +- *(1) LocalTableScan [a#19, b#20, c1#30]
               +- == Initial Plan ==
                  Project [c1#26, c2#27, a#19, b#20]
                  +- BroadcastHashJoin [c1#26], [c1#30], Inner, BuildLeft, false
                     :- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=37]
                     :  +- LocalTableScan [c1#26, c2#27]
                     +- LocalTableScan [a#19, b#20, c1#30]
{noformat}
Bug: Optimize string predicate expressions for UTF8_BINARY_LCASE collation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/24 12:07
Affects Version/s: 4.0.0

Implement {*}contains{*}, {*}startsWith{*}, and *endsWith* built-in string Spark functions using optimized lowercase comparison approach introduced by [~nikolamand-db] in [https://github.com/apache/spark/pull/45816]. Refer to the latest design and code structure imposed by [~uros-db] in https://issues.apache.org/jira/browse/SPARK-47410 to understand how collation support is introduced for Spark SQL expressions. In addition, review previous Jira tickets under the current parent in order to understand how *StringPredicate* expressions are currently used and tested in Spark:
 * [SPARK-47131|https://issues.apache.org/jira/browse/SPARK-47131]
 * [SPARK-47248|https://issues.apache.org/jira/browse/SPARK-47248]
 * [SPARK-47295|https://issues.apache.org/jira/browse/SPARK-47295]

These tickets should help you understand what changes were introduced in order to enable collation support for these functions. Lastly, feel free to use your chosen Spark SQL Editor to play around with the existing functions and learn more about how they work.

 

The goal for this Jira ticket is to improve the UTF8_BINARY_LCASE implementation for the {*}contains{*}, {*}startsWith{*}, and *endsWith* functions so that they use optimized lowercase comparison approach (following the general logic in Nikola's PR), and benchmark the results accordingly. As for testing, the currently existing unit test cases and end-to-end tests should already fully cover the expected behaviour of *StringPredicate* expressions for all collation types. In other words, the objective of this ticket is only to enhance the internal implementation, without introducing any user-facing changes to Spark SQL API.

 

Finally, feel free to refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: StringLPad, StringRPad (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/24 11:57
Affects Version/s: 4.0.0

Enable collation support for the *StringLPad* & *StringRPad* built-in string functions in Spark. First confirm what is the expected behaviour for these functions when given collated strings, then move on to the implementation that would enable handling strings of all collation types. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *StringLPad* & *StringRPad* functions so that they support all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: StringInstr, FindInSet (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/24 11:56
Affects Version/s: 4.0.0

Enable collation support for the *StringInstr* and *FindInSet* built-in string functions in Spark. First confirm what is the expected behaviour for these functions when given collated strings, and then move on to implementation and testing. One way to go about this is to consider using {_}StringSearch{_}, an efficient ICU service for string matching. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *StringInstr* and *FindInSet* functions so that they support all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class, as well as _StringSearch_ using the [ICU user guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html] and [ICU docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html]. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Substring, Right, Left (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/24 12:02
Affects Version/s: 4.0.0

Enable collation support for the *Substring* built-in string function in Spark (including *Right* and *Left* functions). First confirm what is the expected behaviour for these functions when given collated strings, then move on to the implementation that would enable handling strings of all collation types. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the {*}Substring{*}, {*}Right{*}, and *Left* functions so that they support all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Implement python stream writer interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/24 23:20
Affects Version/s: 4.0.0

In order to support developing spark streaming sink in python, we need to implement python stream writer interface.

Reuse PythonPartitionWriter to implement the serialization and execution of write callback in executor.

Implement python worker process to run python streaming data sink committer and communicate with JVM through socket in spark driver. For each python streaming data sink instance there will be a long live python worker process created. Inside the python process, the python write committer will receive abort or commit function call and send back result through socket.
Bug: [K8] Spark App ID collision resulting in deleting wrong resources
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 26/Mar/24 02:18
Affects Version/s: 3.1.0

h3. Issue:

We noticed that sometimes K8s executor pods go in a crash loop. Reason being 'Error: MountVolume.SetUp failed for volume "spark-conf-volume-exec"'. Upon investigation we noticed that there are 2 spark jobs that launched with same application id and when one of them finishes first it deletes all it's resources and deletes the resources of other job too.

-> Spark application ID is created using this [code|https://github.com/apache/spark/blob/36126a5c1821b4418afd5788963a939ea7f64078/core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala#L38]
"spark-application-" + System.currentTimeMillis
This means if 2 applications launch at the same milli second they could end up having same AppId

->  [spark-app-selector|https://github.com/apache/spark/blob/93f98c0a61ddb66eb777c3940fbf29fc58e2d79b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L23] label is added to all resource created by driver and it's value is application Id. Kubernetes Scheduler deletes all the apps with same [label|https://github.com/apache/spark/blob/2a8bb5cdd3a5a2d63428b82df5e5066a805ce878/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L162C1-L172C6] upon termination.

This results in deletion of config map and executor pods of job that's still running, driver tries to relaunch the executor pods, but config map is not present, so it's in crash loop
h3. Context

We are using [Spark of Kubernetes |https://spark.apache.org/docs/latest/running-on-kubernetes.html]and launch our spark jobs using PySpark. We launch multiple Spark Jobs within a given k8s namespace. Each Spark job can be launched from different pods or from different processes in a pod. Every time a job is launched it has a unique app name. Here is how the job is launched (omitting irrelevant details):
{code:java}
# spark_conf has settings required for spark on k8s 
sp = SparkSession.builder \
    .config(conf=spark_conf) \
    .appName('testapp')
sp.master(f'k8s://{kubernetes_host}')
session = sp.getOrCreate()
with session:
    session.sql('SELECT 1'){code}
h3. Repro

Set same app id in spark config, run 2 different jobs, one that finishes fast, one that runs slow. Slower job goes into crash loop
{code:java}
"spark.app.id": "<same Id for 2 spark job>"{code}
h3. Workaround

Set unique spark.app.id for all the jobs that run on k8s

eg:
{code:java}
"spark.app.id": f'{AppName}-{CurrTimeInMilliSecs}-{UUId}'[:63]{code}
h3. Fix

Add unique hash add the end of Application ID: [https://github.com/apache/spark/pull/45712]
Bug: Provide more useful context for PySpark DataFrame API errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/24 00:36
Affects Version/s: 4.0.0

Errors originating from PySpark operations can be difficult to debug with limited context in the error messages. While improvements on the JVM side have been made to offer detailed error contexts, PySpark errors often lack this level of detail. Adding detailed context about the location within the user's PySpark code where the error occurred will help debuggability for PySpark users.