av9ash/GitBugs huggingface dataset:
Bug: Replace mutable.WrappedArray with mutable.ArraySeq
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/23 08:30
Affects Version/s: 4.0.0

{code:java}
@deprecated("Use ArraySeq instead of WrappedArray; it can represent both, boxed and unboxed arrays", "2.13.0")
type WrappedArray[X] = ArraySeq[X]
@deprecated("Use ArraySeq instead of WrappedArray; it can represent both, boxed and unboxed arrays", "2.13.0")
val WrappedArray = ArraySeq {code}
Bug: Website doesn't have horizontal scrollbar
Status: Open | Priority: Major | Resolution: 
Created: 19/Sep/23 01:15
Affects Version/s: 3.5.0

This was reported in the dev mailing list.

- https://lists.apache.org/thread/cfhqgltx0f4flrtb1p5c40zoopdy5yt9

I find a recent issue with the official Spark documentation on the website. Specifically, the Kubernetes configuration lists on the right-hand side are not visible and doc doesn't have a horizontal scrollbar.

 
- [https://spark.apache.org/docs/3.5.0/running-on-kubernetes.html#configuration]
- [https://spark.apache.org/docs/3.4.1/running-on-kubernetes.html#configuration]

Wide tables are broken in the same way.
Bug: No way to exclude jars setting to classpath while doing spark-submit
Status: Open | Priority: Major | Resolution: 
Created: 11/Sep/23 12:19
Affects Version/s: 3.4.1

The challenge is whenever you do spark-submit to start the application, the jars present in spark home directory gets added to classpath automatically and there is no way to exclude specific jars from there. For example, we dont want slf4j jars present in spark home directory to be setted in classpath as in codebase slf4j is already there. Thus it causes conflicts in jars. This forces user to change there codebase to support spark-submit or to manually remove the jars from spark-home directory. This i believe is not right practice as we deviating from using spark as it supposed to be and it causes unfixable behaviors at various instances with no clue. Example linkages errors are common with the jar conflicts. 

 

There is detailed stackoverflow question on this issue. 

refer : https://stackoverflow.com/questions/76476618/linkageerror-facing-while-doing-spark-submit
Bug: Improve ClosureCleaner to support closures defined in Ammonite REPL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/23 14:13
Affects Version/s: 3.5.1

ConnectRepl uses Ammonite REPL with  CodeClassWrapper to run Scala code. It means that each code cell is wrapped into a separate object. If there are multiple variables defined in the same cell / code block it will lead to capturing extra variables, increasing serialized UDF payload size or making it non-serializable.

For example, this code
{code:java}
// cell 1 
{
  val x = 100
  val y = new NonSerializable
}

// cell 2
spark.range(10).map(i => i + x).agg(sum("value")).collect(){code}
will fail because lambda will capture both `x` and `y` as they're defined in the same wrapper object
Bug: Assign name to _LEGACY_ERROR_TEMP_2151
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Sep/23 08:47
Affects Version/s: 3.5.0

in DatasetSuite test("CLASS_UNSUPPORTED_BY_MAP_OBJECTS when creating dataset") , we are using _LEGACY_ERROR_TEMP_2151, We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Fix IDENTIFIER clause for functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/23 09:24
Affects Version/s: 3.5.0

Due to a quirk in the grammar IDENTIFIER('foo')(<somearg>) does not resolve depending on <somearg>.

Example:
SELECT IDENTIFIER('abs')(-1) works, but
SELECT IDENTIFIER('abs')(c1) FROM VALUES(-1) AS T(c1) does not.
Bug: Support stage level task resource profile for yarn cluster when dynamic allocation disabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/23 11:50
Affects Version/s: 3.4.1

[https://github.com/apache/spark/pull/37268] has introduced a new feature that supports stage-level schedule task resource profile for standalone cluster when dynamic allocation is disabled. It's really cool feature, especially for ML/DL cases, more details can be found in that PR.

 

The problem here is that the feature is only available for standalone cluster for now, but most users would also expect it can be used for other spark clusters like yarn and k8s.

 

So I file this issue to track this task.
Bug: Since version 3.2.0, Spark SQL has taken longer to execute "show paritions",probably because of changes introduced by  SPARK-35278
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/23 16:58
Affects Version/s: 3.2.0

After SPARK-35378 was changed, I noticed that the execution of statements such as ‘show parititions test' became slower.

The change point is that the execution process changes from ExecutedCommandEnec to CommandResultExec, but ExecutedCommandExec originally implemented the following method

override def executeToIterator(): Iterator[InternalRow] = sideEffectResult.iterator

CommandResultExec is not rewritten, so when the hasNext method is executed, a job process is created, resulting in increased time-consuming
Bug: Missing case for RelationTimeTravel in CheckAnalysis
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/23 19:40
Affects Version/s: 3.5.0

{{CheckAnalysis.checkAnalysis0}} lacks a case for {{{}RelationTimeTravel{}}}, and since the latter is (intentionally) an {{UnresolvedLeafNode}} rather than a {{{}UnaryNode{}}}, the existing checks do not traverse it.

Result: Attempting time travel over a non-existing table produces a spark internal error from the [default case|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L818], rather than the expected {{{}AnalysisException{}}}:
{code:java}
[info]   Cause: org.apache.spark.SparkException: [INTERNAL_ERROR] Found the unresolved operator: 'RelationTimeTravel 'UnresolvedRelation [not_exists], [], false, 0
[info]   at org.apache.spark.SparkException$.internalError(SparkException.scala:77)
[info]   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$54(CheckAnalysis.scala:753) {code}
Fix should be simple enough:
{code:java}
case tt: RelationTimeTravel =>
  checkAnalysis0(tt.table) {code}
Bug: Deprecate spark.sql.parser.escapedStringLiterals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/23 08:24
Affects Version/s: 4.0.0

The config allows to switch to legacy behaviour of Spark 1.6 which is pretty old and not maintained anymore. Deprecation and removing of the config in the future versions should improve code maintenance. Also there is an alternative approach by using RAW string literals in which Spark doesn't especially handle escaped character sequences.
Bug: Improve the description for `LIKE`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/23 20:30
Affects Version/s: 4.0.0

The description of `LIKE` says:
{code}
... in order to match "\abc", the pattern should be "\\abc"
{code}
but in Spark SQL shell:
{code:sql}
spark-sql (default)> SELECT c FROM t;
\abc
spark-sql (default)> SELECT c LIKE "\\abc" FROM t;
[INVALID_FORMAT.ESC_IN_THE_MIDDLE] The format is invalid: '\\abc'. The escape character is not allowed to precede 'a'.
spark-sql (default)> SELECT c LIKE "\\\\abc" FROM t;
true
{code}
Bug: Describe characters unescaping in string literals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/23 18:16
Affects Version/s: 4.0.0

Update the page https://spark.apache.org/docs/latest/sql-ref-literals.html#string-literal and describe the escaping implemented at https://github.com/apache/spark/blob/9109d7037f44158e72d14019eb33f9c7b8838868/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala#L38
Bug: Replcace `TraversableOnce` with `IterableOnce`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/23 04:13
Affects Version/s: 4.0.0

{code:java}
@deprecated("Use IterableOnce instead of TraversableOnce", "2.13.0")
type TraversableOnce[+A] = scala.collection.IterableOnce[A]

type IterableOnce[+A] = scala.collection.IterableOnce[A] {code}
Bug: Incorrect COUNT bug handling in scalar subqueries
Status: Open | Priority: Major | Resolution: 
Created: 29/Sep/23 16:04
Affects Version/s: 3.5.0

This query has incorrect results:
create temp view l (a, b)
as values
(1, 2.0),
(1, 2.0),
(2, 1.0),
(2, 1.0),
(3, 3.0),
(null, null),
(null, 5.0),
(6, null);

create temp view r (c, d)
as values
(2, 3.0),
(2, 3.0),
(3, 2.0),
(4, 1.0),
(null, null),
(null, 5.0),
(6, null);

select (
select sum(cnt)
from (select count ( * ) cnt from r where l.a = r.c)
) from l;
 
 
It returns 

– !query output
1
1
2
2
NULL
NULL
NULL
NULL

NULLs in the output should be zeros.
Bug: Deadlock caused by rdd replication level of 2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/23 23:32
Affects Version/s: 3.4.1

 
When 2 tasks try to compute same rdd with replication level of 2 and running on only 2 executors. Deadlock will happen.

Task only release lock after writing into local machine and replicate to remote executor.

 
||Time||Exe 1 (Task Thread T1)||Exe 1 (Shuffle Server Thread T2)||Exe 2 (Task Thread T3)||Exe 2 (Shuffle Server Thread T4)||
|T0|write lock of rdd| | | |
|T1| | |write lock of rdd| |
|T2|replicate -> UploadBlockSync (blocked by T4)| | | |
|T3| | | |Received UploadBlock request from T1 (blocked by T4)|
|T4| | |replicate -> UploadBlockSync (blocked by T2)| |
|T5| |Received UploadBlock request from T3 (blocked by T1)| | |
|T6|Deadlock|Deadlock|Deadlock|Deadlock|
Bug: Resolve functions bottom-up
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/23 10:21
Affects Version/s: 4.0.0

This PR proposes bottum-up resolution in {{{}ResolveFunctions{}}}, which is much faster if we have deeply nested {{{}UnresolvedFunctions{}}}. These structures are more likely to occur after [#42864|https://github.com/apache/spark/pull/42864].
Bug: Respect `spark.sql.files.ignoreMissingFiles` in HadoopRDD
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/23 13:08
Affects Version/s: 4.0.0

Currently, the SQL config spark.sql.files.ignoreMissingFiles influences on RDDs created in Spark SQL such as FileScanRDD but doesn't impact on HadoopRDD and NewHadoopRDD. The last RDDs have separate core config spark.files.ignoreMissingFiles. That inconsistency might confuse users if they don't know implementation details. This ticket aims to eliminate the inconsistency.
Bug: The driver did not receive the StatusUpdate sent by the executor
Status: Open | Priority: Major | Resolution: 
Created: 26/Sep/23 01:05
Affects Version/s: 2.4.0

I have a spark task that occasionally runs for a long time and does not finished. Usually it is normal and I did not find any error logs. The deployment mode of the task is standalone
But I found from the driver and executor logs that the executor showed that some tasks had been completed, but the driver only received one StatusUpdate event.
Bug: pyspark 3.5.0 missing in pypi
Status: Open | Priority: Major | Resolution: 
Created: 25/Sep/23 20:18
Affects Version/s: 3.5.0

the pyspark 3.5.0 is not present in pypi -> [https://pypi.org/project/pyspark/#history]
the version is 3.4.1 is currently the latest
Bug: SHOW COLUMNS namespace is not catalog-aware 
Status: Open | Priority: Major | Resolution: 
Created: 25/Sep/23 15:08
Affects Version/s: 3.5.0

Today, the namespace argument of {{SHOW COLUMNS}} command is only partly catalog-aware. [AstBuilder.scala|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala#L4693] correctly pulls in and applies namespaces with more than one element, but [ResolveSessionCatalog.scala|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala#L316] does not correctly validate a multi-element namespace against the qualified table name:
{code:java}
    case ShowColumns(ResolvedV1TableOrViewIdentifier(ident), ns, output) =>
      val v1TableName = ident
      val resolver = conf.resolver
      val db = ns match {
        case Some(db) if v1TableName.database.exists(!resolver(_, db.head)) =>
          throw QueryCompilationErrors.showColumnsWithConflictDatabasesError(db, v1TableName)
        case _ => ns.map(_.head)
      } {code}
By always/only checking {{db.head}} against {{{}v1TableName.database{}}}, it will not correctly handle e.g.
{code:java}
SHOW COLUMNS FOR a.b.table IN a.b -- incorrectly fails
{code}
There is also some ambiguity in the semantics of the check:
{code:java}
SHOW COLUMNS FOR a.b.table IN a -- fails today (should it succeed?) 
SHOW COLUMNS FOR a.b.table IN b -- succeeds today (should it fail?) {code}
It might work better to use an actual {{Identifier}} (instead of {{{}TableIdentifier{}}}), and compare its {{namespace}} against the user-provided namespace?

Tangentially: It's a arguably a bit strange for the validity check to reside in {{ResolveSessionCatalog}} rule, given that it doesn't actually have anything to do with session catalogs? This seems more an artifact of an implementation that searches specifically for V1 tables, and {{ResolveSessionCatalog}} providing matchers that make that job easier?
Bug: Remove the workaround for JDK-8228469 in SPARK-31959 test
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/23 02:59
Affects Version/s: 4.0.0

In https://issues.apache.org/jira/browse/SPARK-31959, we added a bit of workaround for outdated timezone in the tests. We can now remove them because we dropped JDK 11 at SPARK-44112
Bug: Make `InMemoryColumnarBenchmark` use AQE-aware utils to collect plans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/23 05:40
Affects Version/s: 3.5.1

After SPARK-42768, the default value of `spark.sql.optimizer.canChangeCachedPlanOutputPartitioning` has changed from false to true, so we should use AQE-aware utils to collect plans.
Bug: Comment out unused JDK 11 related in dev/run-tests.py
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/23 02:48
Affects Version/s: 4.0.0

{code}
    # set up java11 env if this is a pull request build with 'test-java11' in the title
    if "ghprbPullTitle" in os.environ:
        if "test-java11" in os.environ["ghprbPullTitle"].lower():
            os.environ["JAVA_HOME"] = "/usr/java/jdk-11.0.1"
            os.environ["PATH"] = "%s/bin:%s" % (os.environ["JAVA_HOME"], os.environ["PATH"])
            test_profiles += ["-Djava.version=11"]
{code}

we don't need this anymore
Bug: replace function fails to handle null replace param
Status: Open | Priority: Major | Resolution: 
Created: 22/Sep/23 07:19
Affects Version/s: 3.4.0

[replace |https://spark.apache.org/docs/latest/api/sql/#replace]function fails to handle null replace param, example below:

 
df.withColumn('test',F.expr('replace(col1, "nUll", 1)')).show()
|| ||col1||col2||test||
||0|person1|0.0|person1|
||1|person1|2.0|person1|
||2|person1|3.0|person1|
||3|person2|1.0|person2|
||4|None|2.0|None|
||5|nUll|None|1|

 
df.withColumn('test',F.expr('replace(col1, "nUll", null)')).show()
|| ||col1||col2||test||
||0|person1|0.0|None|
||1|person1|2.0|None|
||2|person1|3.0|None|
||3|person2|1.0|None|
||4|None|2.0|None|
||5|nUll|None|None|

 

 

This function has been ported over to 3.5.0 but I've not been able to test it on that yet
Bug: Configurable error when generating Python docs
Status: Open | Priority: Major | Resolution: 
Created: 21/Sep/23 22:04
Affects Version/s: 4.0.0

{{cd python/docs}}

{{make html }}

 

Gives a Configuration error:
There is a programmable error in your configuration file:

ImportError: Warning: Latest version of pandas (2.1.0) is required to generate the documentation; however, your version was 2.0.3

make: *** [html] Error 2
Bug: Why isCheckpointed returns false while as checkpointing is eager by default?
Status: Open | Priority: Major | Resolution: 
Created: 21/Sep/23 09:28
Affects Version/s: 3.3.0

Hi!

I consider why `isCheckpointed` method return `false` - could someone explain it? I would expect `true`.
{code:java}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.0
      /_/
         
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.13)
Type in expressions to have them evaluated.
Type :help for more information.
scala> sc.setCheckpointDir("file:///tmp/")
scala> val df1 = Seq(1,2,3,4).toDF
df1: org.apache.spark.sql.DataFrame = [value: int]
scala> val df2 = df1.checkpoint()
df2: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int]
scala> df2.rdd.isCheckpointed
res2: Boolean = false  // why false?{code}
Bug: The `sql()` method doesn't support map and array parameters in Python connect client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/23 11:18
Affects Version/s: 4.0.0

At the moment, the Python connect client fails when we pass a map or an array as a parameter to the `sql()` method. This ticket aims to fix the behaviour to reach feature parity with the regular PySpark.
Bug: Define a new error class and apply for the case where checkpointing state to DFS fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/23 21:56
Affects Version/s: 4.0.0

[From Neil Ramswamy]:

When there is an exception during storing state from DFS (Hadoop, object store, etc), we just propagate the exception without the context. There is a context if customers look into stack trace, but that is definitely not something we want customers to look into by themselves.

For example, let’s say file system related exception happened during checkpointing state. Since we just let the exception be bubbled up to the very top without adding any context, it is quite confusing for customers to quickly indicate whether there is an issue with source/sink (if source/sink data source is based on file), or offset/commit log, or state load/checkpoint. Too many operations can throw the same exception.

The ticket aims to wrap the exception during the commit of the state, to assign error class properly. With assigning error class, we can classify the errors which help us to determine what errors customers are struggling much.

StateStore.commit() is the entry point. Each state store provider has its own implementation, but use the same error class across implementations, as we want to categorize it as the same. Even better if we can put it to the higher-level of caller, but would be OK to handle it in built-in impls.
Bug: Fallback to use single batch executor for Trigger.AvailableNow with unsupported sources rather than using wrapper
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/23 04:52
Affects Version/s: 4.0.0

We have observed the case where wrapper implementation of Trigger.AvailableNow (
AvailableNowDataStreamWrapper and subclasses) is not fully compatible with 3rd party data source and brought up correctness issue.
 
While we could persuade 3rd party data source to support Trigger.AvailableNow, pursuing all 3rd parties to do this is too aggressive and challenging goal we never be able to make. Also, it may not be also possible to come up with the wrapper implementation which would have zero issue with any arbitrary source.
 
As a mitigation, we want to make a slight behavioral change for such case, falling back to single batch execution (a.k.a. Trigger.Once) rather than using wrapper implementation. The exact behavior between Trigger.AvailableNow and Trigger.Once are different so it's technically behavioral change, but it's probably lot less surprised than failing the query.
 
For extreme case where users are confident that there will be no issue at all on using wrapper, we will come up with a flag to provide the previous behavior.
Bug: Correct the default value of `spark.history.store.hybridStore.diskBackend` in `monitoring.md`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/23 12:36
Affects Version/s: 3.4.2

SPARK-42277 change to use RocksDB for spark.history.store.hybridStore.diskBackend by default, but in `monitoring.md`, the default value is still set as LEVELDB.
Bug: Add Unicode[encode/decode]() function
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/23 12:42
Affects Version/s: 3.5.0

unicode_encode()/unicode_decode() is a common convenient function supported by a number of database systems. it can help with data analysis efficiency

 
{code:java}
//代码占位符
select unicode_encode('apache');
return \u0061\u0070\u0061\u0063\u0068\u0065

select unicode_decode('\u0061\u0070\u0061\u0063\u0068\u0065');
return apache{code}
Bug: Update `test_axis_on_dataframe` when Pandas regression is fixed
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/23 03:39
Affects Version/s: 4.0.0

We manually cast the datatype when testing `test_axis_on_dataframe` from [https://github.com/apache/spark/pull/43002,|https://github.com/apache/spark/pull/43002.] but it's not a normal way to test properly.

After the regression of Pandas is resolved, we should return the test back to normal way.

See Pandas regression: https://github.com/pandas-dev/pandas/issues/55194
Bug: Columns should not be visible for filter after projection
Status: Open | Priority: Major | Resolution: 
Created: 19/Sep/23 08:48
Affects Version/s: 3.3.1

Columns are visible for filtering but not for select after projection. Moreover the behaviour is different when after a union (in this case columns are not visible for filtering anymore).
{code:java}
from pyspark.sql import SparkSession
from pyspark.sql.types import *

data1 = []
data2 = []

for i in range(2): 
    data1.append( (1,i) )
    data2.append( (2,i+10))



schema1 = StructType([
        StructField('f1', IntegerType(), True),
         StructField('f2', IntegerType(), True)
])



df1 = spark.createDataFrame(data1, schema1)
df2 = spark.createDataFrame(data2, schema1)


df1.show()
df2.show()


#works, f1 is available for filter (though it should not be)
df1.select('f2').where('f1=1').show()

#error, f1 is not available
df1.select('f2').union(df2.select('f2')).where('f1=1').show()

#this is semantically not symmetric -> incorrect. 

{code}

This is similar to this one: https://issues.apache.org/jira/browse/SPARK-30421
Perhaps it gives a bit more argumentation why this should be fixed as it is logically not correct.
Bug: Install independent Python linter dependencies for branch-3.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Sep/23 04:12
Affects Version/s: 4.0.0

Python linter failed in branch -3.5 daily test:
 * [https://github.com/apache/spark/actions/runs/6221638911/job/16884068430]

{code:java}
Run PYTHON_EXECUTABLE=python3.9 ./dev/lint-python
12starting python compilation test...
13python compilation succeeded.
14
15starting black test...
16black checks failed:
17Oh no! 💥 💔 💥 The required version `22.6.0` does not match the running version `23.9.1`!
18Please run 'dev/reformat-python' script.
191
20Error: Process completed with exit code 1. {code}
Bug: Switch languages consistently across docs for all code snippets (Spark 3.4 and below)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Sep/23 02:56
Affects Version/s: 3.1.3

Similar with SPARK-44820 but needs different change as they were refactored at https://github.com/apache/spark/commit/12b9b771c7ad75cb90c0a51cd2d0581dd3c719e2