av9ash/GitBugs huggingface dataset:
Bug: spark-connect-repl is not working on macOS
Status: Open | Priority: Major | Resolution: 
Created: 01/Oct/23 08:39
Affects Version/s: 3.5.0

I followed [https://spark.apache.org/docs/latest/spark-connect-overview.html#use-spark-connect-for-interactive-analysis] to try spark-connect-repl on my local PC but got the following error:

 

---------------------------

spark-connect-repl
Exception in thread "main" java.lang.NoClassDefFoundError: org/sparkproject/connect/client/com/google/common/io/BaseEncoding
    at org.sparkproject.connect.client.io.grpc.Metadata.<clinit>(Metadata.java:114)
    at org.apache.spark.sql.connect.client.SparkConnectClient$.<init>(SparkConnectClient.scala:329)
    at org.apache.spark.sql.connect.client.SparkConnectClient$.<clinit>(SparkConnectClient.scala)
    at org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:61)
    at org.apache.spark.sql.application.ConnectRepl$.main(ConnectRepl.scala:50)
    at org.apache.spark.sql.application.ConnectRepl.main(ConnectRepl.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at coursier.bootstrap.launcher.a.a(Unknown Source)
    at coursier.bootstrap.launcher.Launcher.main(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.sparkproject.connect.client.com.google.common.io.BaseEncoding
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 12 more

---------------------------
 
Do you have any idea why this is happening and how to solve it? 
Thank you.
Bug: Include `ESCAPE` to `sql()` of `Like`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/23 13:02
Affects Version/s: 4.0.0

Fix the `sql()` method of the `Like` expression and append the `ESCAPE` closure. That should become consistent to `toString` and fix the issue:

{code:sql}
spark-sql (default)> create temp view tbl as (SELECT 'a|_' like 'a||_' escape '|', 'a|_' like 'a||_' escape 'a');
[COLUMN_ALREADY_EXISTS] The column `a|_ like a||_` already exists. Consider to choose another name or rename the existing column.
{code}
Bug: RocksDB State Store Doesn't Need to Recheck checkpoint path existence
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Oct/23 17:56
Affects Version/s: 3.5.1

Right now, every time RocksDB.load() is called, we check checkpoint directory existence and create it if not. This is relatively expensive and show up in performance profiling. We don't need to do it the second time.
Bug: okio-1.15.0 CVE-2023-3635
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Oct/23 06:49
Affects Version/s: 3.4.0

CVE-2023-3635 is being flagged against okio-1.15.0 present in the Spark 3.5.0 build:
 * ./spark-3.5.0-bin-without-hadoop/jars/okio-1.15.0.jar
 * ./spark-3.5.0-bin-hadoop3/jars/okio-1.15.0.jar

I don't see okio in the dependency tree, it must be coming in via some profile.
Bug: Avoid use spark.yarn.user.classpath.first
Status: Open | Priority: Minor | Resolution: 
Created: 30/Oct/23 06:35
Affects Version/s: 3.5.0

When we set spark.yarn.user.classpath.first=true, we get the following message

 
{code:java}
23/10/30 14:32:16.855 pool-1-thread-1-ScalaTest-running-ClientSuite WARN SparkConf: The configuration key 'spark.yarn.user.classpath.first' has been deprecated as of Spark 1.3 and may be removed in the future. Please use spark.{driver,executor}.userClassPathFirst instead. {code}
Bug: Migrate jetty 9 to jetty 10
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Oct/23 13:31
Affects Version/s: 4.0.0

Jetty 12 supports JakartaEE 8/JakartaEE 9/JakartaEE 10 simultaneously. But the version span is quite large, need to read the documentation in detail, not sure if it can be completed within the 4.0 cycle, so it's set to low priority.
Bug: Inconsistent status code between web page and REST API when exception is thrown
Status: Open | Priority: Minor | Resolution: 
Created: 16/Oct/23 14:26
Affects Version/s: 3.5.0

Spark history server provides [AppHistoryServerPlugin|https://github.com/kuwii/spark/blob/dev/status-code/core/src/main/scala/org/apache/spark/status/AppHistoryServerPlugin.scala] to add extra REST API and web pages. However there's an issue when exceptions are thrown, causing incnosistent status code between web page and REST API.

For REST API, if the thrown exception is an instance of WebApplicationException, then the status code will be set as the one defined within the exception.

However for web page, all exceptions are wrapped within a 500 response.
Bug: Remove redundant"Auto-application to `()` is deprecated" compile suppression rules.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Oct/23 10:44
Affects Version/s: 4.0.0

Due to the issue https://github.com/scalatest/scalatest/issues/2297, we need to wait until we upgrade a scalatest version before removing these suppression rules.

Maybe 3.2.18
Bug: Improve the assert message in `RollingEventLogFilesFileReader`
Status: Open | Priority: Minor | Resolution: 
Created: 26/Oct/23 03:04
Affects Version/s: 3.5.0

Currently the assert message in `RollingEventLogFilesFileReader` is not clear. When the assertion fails, it's difficult to find the event log file that is having problems.

In this ticket, we will update the following two places to make the assert message clearer.
 * {*}val files{*}. Add `rootPath{*}`{*} field in the assert message.
 * {*}val eventLogFiles{*}. Add new method `findMissingIndices`, only print the missing event log indices instead of all indices.
Bug: codegen sum(decimal_column / 2) computes div twice
Status: Open | Priority: Minor | Resolution: 
Created: 10/Oct/23 07:15
Affects Version/s: 3.2.0

After eliminates all common subexpression of `If`, it reduces computation time from 93.727s to 55.498s over 15393291 records. 

*The SQL to reproduce the result*
{code:java}
create table t_dec (c1 decimal(6,2));
insert into t_dec values(1.0),(2.0),(null),(3.0);
explain codegen select sum(c1/2) from t_dec; {code}
 
*Reasons may cause the result:* 
 
Function sum use If expression in updateExpressions:
 `If(child.isNull, sum, sum + KnownNotNull(child).cast(resultType))`
 
The three variables in if expression like this.
{code:java}
predicate: isnull(CheckOverflow((promote_precision(input[2, decimal(10,0), true]) / 2), DecimalType(16,6), true))trueValue: input[0, decimal(26,6), true]falseValue: (input[0, decimal(26,6), true] + cast(knownnotnull(CheckOverflow((promote_precision(input[2, decimal(10,0), true]) / 2), DecimalType(16,6), true)) as decimal(26,6))) {code}
In sub expression elimination, only predicate is evaluated in EquivalentExpressions# childrenToRecurse
{code:java}
private def childrenToRecurse(expr: Expression): Seq[Expression] = expr match {
  case _: CodegenFallback => Nil
  case i: If => i.predicate :: Nil
  case c: CaseWhen => c.children.head :: Nil
  case c: Coalesce => c.children.head :: Nil
  case other => other.children
} {code}
I tried to replace `case i: If => i.predicate :: Nil` with 'case i: If => i.predicate :: trueValue :: falseValue :: Nil', and it produce correct result.

 

But the following comment in `childrenToRecurse`  makes me not sure it will cause any other problems.
{code:java}
// 2. If: common subexpressions will always be evaluated at the beginning, but the true and
// false expressions in `If` may not get accessed, according to the predicate
// expression. We should only recurse into the predicate expression. {code}
Bug:    Replace `s.c.MapOps.mapValues` with `s.c.MapOps.view.mapValues`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Oct/23 07:51
Affects Version/s: 4.0.0

{code:java}
@deprecated("Use .view.mapValues(f). A future version will include a strict version of this method (for now, .view.mapValues(f).toMap).", "2.13.0")
def mapValues[W](f: V => W): MapView[K, W] = new MapView.MapValues(this, f) {code}
Bug: Fix `method signum in trait ScalaNumberProxy is deprecated`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Oct/23 02:34
Affects Version/s: 4.0.0

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scalalang:194:25: method signum in trait ScalaNumberProxy is deprecated (since 2.13.0): use `sign` method instead
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalyst.expressions.EquivalentExpressions.updateExprTree.uc, origin=scala.runtime.ScalaNumberProxy.signum, version=2.13.0
[warn]       val uc = useCount.signum
[warn]   {code}
Bug: Fix `method force in trait View is deprecated`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Oct/23 02:35
Affects Version/s: 4.0.0

{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:368:36: method force in trait View is deprecated (since 2.13.0): Views no longer know about their underlying collection type; .force always returns an IndexedSeq
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalyst.trees.TreeNode.legacyWithNewChildren.newArgs.$anonfun, origin=scala.collection.View.force, version=2.13.
[warn]         m.mapValues(mapChild).view.force.toMap
[warn]                                    ^ {code}
Bug: Cover BufferReleasingInputStream.available under tryOrFetchFailedException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Oct/23 21:00
Affects Version/s: 3.4.1

We have encountered shuffle data corruption issue:

```
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
	at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:112)
	at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
	at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:504)
	at org.xerial.snappy.Snappy.uncompress(Snappy.java:543)
	at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:450)
	at org.xerial.snappy.SnappyInputStream.available(SnappyInputStream.java:497)
	at org.apache.spark.storage.BufferReleasingInputStream.available(ShuffleBlockFetcherIterator.scala:1356)
 ```

Spark shuffle has capacity to detect corruption for a few stream op like `read` and `skip`, such `IOException` in the stack trace will be rethrown as `FetchFailedException` that will re-try the failed shuffle task. But in the stack trace it is `available` that is not covered by the mechanism. So no-retry has been happened and the Spark application just failed.

As the `available` op will also involve data decompression, we should be able to check it like `read` and `skip` do.
Bug: Use java.lang.ref.Cleaner instead of org.apache.spark.sql.connect.client.util.Cleaner
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Oct/23 18:14
Affects Version/s: 4.0.0

Now, we have updated JDK to 17,  so should replace this class by [[java.lang.ref.Cleaner]].

 

!image-2023-10-19-02-25-57-966.png!
Bug: Add `since` field to Java API marked as `@Deprecated`.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Oct/23 06:04
Affects Version/s: 4.0.0

Spark 3.0.0:
- SPARK-26861
  - org.apache.spark.sql.expressions.javalang.typed
- SPARK-27606
  - org.apache.spark.sql.catalyst.expressions.ExpressionDescription#extended: 
  - org.apache.spark.sql.catalyst.expressions.ExpressionInfo#ExpressionInfo(String, String, String, String, String)

Spark 3.2.0
- SPARK-33717
  - org.apache.spark.launcher.SparkLauncher#DEPRECATED_CHILD_CONNECTION_TIMEOUT
- SPARK-33779
  - org.apache.spark.sql.connector.write.WriteBuilder#buildForBatch
  - org.apache.spark.sql.connector.write.WriteBuilder#buildForStreaming
Spark 3.4.0
- SPARK-39805
  - org.apache.spark.sql.streaming.Trigger
- SPARK-42398
  - org.apache.spark.sql.connector.catalog.TableCatalog#createTable(Identifier, StructType, Transform[], Map<String,String>) 
  - org.apache.spark.sql.connector.catalog.StagingTableCatalog#stageCreate(Identifier, StructType, Transform[], Map<String,String>)
  - org.apache.spark.sql.connector.catalog.Table#schema
Bug: Re-use Literal objects when replacing timestamps in the ComputeCurrentTime rule
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Oct/23 06:12
Affects Version/s: 3.5.0

The ComputeCurrentTime optimizer rule does produce unique timestamp Literals for current time expressions of a query. For CurrentDate and LocalTimestamp objects only the literal objects are not re-used though, but equal objects are created for each instance. This can cost unnecessary much memory in case there are many such Literal objects.
Bug: RocksDB State Store Should Lower RocksDB Background Thread CPU Priority
Status: Open | Priority: Minor | Resolution: 
Created: 11/Oct/23 23:00
Affects Version/s: 3.5.1

We can move RocksDB flush and compaction to lower CPU priority. They usually are background tasks and don't need to compete with task execution. For the case where a task may wait for some RocksDB background task to finish, such as checkpointing, or waiting async checkpointing to finish, the task slot is waiting so we are likely to have enough CPU for low pri CPU anyway.
Bug: RocksDB State Store to Use LZ4 Compression
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Oct/23 22:47
Affects Version/s: 3.5.1

LZ4 is generally faster than Snappy. That's probably we use LZ4 in changelogs and other places by default. However, we don't change RocksDB's default of Snappy compression style. The RocksDB Team recommend LZ4 or ZSTD and the default is kept to Snappy only for backward compatible reason. We should use LZ4 instead.
Bug: Usages of ParVector are unsafe because it does not propagate ThreadLocals or SparkSession
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Oct/23 17:41
Affects Version/s: 3.5.0

CastSuiteBase and ExpressionInfoSuite use ParVector.foreach() to run Spark SQL queries in parallel. They incorrectly assume that each parallel operation will inherit the main thread’s active SparkSession. This is only true when these parallel operations run in freshly-created threads. However, when other code has already run some parallel operations before Spark was started, then there may be existing threads that do not have an active SparkSession. In that case, these tests fail with NullPointerExceptions when creating SparkPlans or running SQL queries.

The fix is to use the existing method ThreadUtils.parmap(). This method creates fresh threads that inherit the current active SparkSession, and it propagates the Spark ThreadLocals.

We should also add a scalastyle warning against use of ParVector.
Bug: spark.python.pyspark.sql.functions Typo at date_format Function
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Oct/23 17:42
Affects Version/s: 3.5.0

In the spark.python.pyspark.sql.functions module, at the {{date_format}} method's doctest, there is a typo in the year format.

Instead of '{{{}MM/dd/yyy'{}}}, it should be {{'MM/dd/yyyy'}} as the expected output{{{}[Row(date='04/08/2015')]{}}} indicates the following format {{"MM/dd/yyyy".}}

 

From the official documentation:

!image-2023-10-19-19-46-22-918.png|width=633,height=365!
{code:java}
df = spark.createDataFrame([('2015-04-08',)], ['dt'])
df.select(date_format('dt', 'MM/dd/yyy').alias('date')).collect()
[Row(date='04/08/2015')]
{code}
 

As a solution, I proposed the PR [https://github.com/apache/spark/pull/43442].
Bug: Replace `setSafeMode(HdfsConstants.SafeModeAction, boolean)` with `setSafeMode(SafeModeAction, boolean)`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Oct/23 10:13
Affects Version/s: 4.0.0

{code:java}
/**
 * Enter, leave or get safe mode.
 *
 * @param action
 *          One of SafeModeAction.ENTER, SafeModeAction.LEAVE and
 *          SafeModeAction.GET.
 * @param isChecked
 *          If true check only for Active NNs status, else check first NN's
 *          status.
 *
 * @see org.apache.hadoop.hdfs.protocol.ClientProtocol#setSafeMode(HdfsConstants.SafeModeAction,
 * boolean)
 *
 * @deprecated please instead use
 *               {@link DistributedFileSystem#setSafeMode(SafeModeAction, boolean)}.
 */
@Deprecated
public boolean setSafeMode(HdfsConstants.SafeModeAction action,
    boolean isChecked) throws IOException {
  return dfs.setSafeMode(action, isChecked);
} {code}
 

`setSafeMode(HdfsConstants.SafeModeAction, boolean)` is `Deprecated`
Bug: Fix compilation warnings related to other-nullary-override
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Oct/23 07:58
Affects Version/s: 4.0.0

{code:java}
[error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/CloseableIterator.scala:36:16: method with a single empty parameter list overrides method hasNext in trait Iterator defined without a parameter list [quickfixable]
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=other-nullary-override, site=org.apache.spark.sql.connect.client.WrappedCloseableIterator
[error]   override def hasNext(): Boolean = innerIterator.hasNext
[error]                ^
[error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ExecutePlanResponseReattachableIterator.scala:136:16: method without a parameter list overrides method hasNext in class WrappedCloseableIterator defined with a single empty parameter list [quickfixable]
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=other-nullary-override, site=org.apache.spark.sql.connect.client.ExecutePlanResponseReattachableIterator
[error]   override def hasNext: Boolean = synchronized {
[error]                ^
[error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:73:20: method without a parameter list overrides method hasNext in class WrappedCloseableIterator defined with a single empty parameter list [quickfixable]
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=other-nullary-override, site=org.apache.spark.sql.connect.client.GrpcExceptionConverter.convertIterator
[error]       override def hasNext: Boolean = {
[error]                    ^
[error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala:77:18: method without a parameter list overrides method next in class WrappedCloseableIterator defined with a single empty parameter list [quickfixable]
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=other-nullary-override, site=org.apache.spark.sql.connect.client.GrpcRetryHandler.RetryIterator
[error]     override def next: U = {
[error]                  ^
[error] /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala:81:18: method without a parameter list overrides method hasNext in class WrappedCloseableIterator defined with a single empty parameter list [quickfixable]
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=other-nullary-override, site=org.apache.spark.sql.connect.client.GrpcRetryHandler.RetryIterator
[error]     override def hasNext: Boolean = {
[error]                  ^
 {code}
Bug: pyspark connect overwrite_partitions bug
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Oct/23 09:06
Affects Version/s: 3.5.0

DataFrameWriterV2.overwritePartitions set mode as *overwrite_partitions* [pyspark/sql/connect/readwriter.py, line 825], but WirteOperationV2 take it as *overwrite_partition* [pyspark/sql/connect/plan.py, line 1660]
Bug: Followup: Ignore task completion from old stage after retrying indeterminate stages
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Oct/23 06:30
Affects Version/s: 3.5.1

With SPARK-45182, we added a fix for not letting laggard tasks of the older attempts of the indeterminate stage from marking the partition has completed in map output tracker.

When a task completes, DAG scheduler also notifies all the tasksets of the stage about that partition being completed. Tasksets would not schedule such task if they are not already scheduled. This is not correct for indeterminate stage, since we want to re-run all the tasks on re-attempt
Bug: Use pattern matching for type checking and conversion
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Oct/23 07:08
Affects Version/s: 4.0.0

Refer to [JEP 394|https://openjdk.org/jeps/394]

Example:
{code:java}
if (obj instanceof String) {
    String str = (String) obj;
    System.out.println(str);
} {code}

Can be replaced with

 
{code:java}
if (obj instanceof String str) {
    System.out.println(str);
} {code}
The new code look more compact
Bug: cleanSource problem on FileStreamSource for Windows env
Status: Open | Priority: Minor | Resolution: 
Created: 12/Oct/23 12:52
Affects Version/s: 3.4.1

We are using Spark with Scala in Windows environment. While streaming using Spark, I give the *{{cleanSource}}* option as "archive" and the *{{sourceArchiveDir}}* option as "archived" as in the code below.
{code:java}
spark.readStream
  .option("cleanSource", "archive")
  .option("sourceArchiveDir", "archived"){code}
When I tried this in a Linux environment, I realized that the problem was with the paths. Because when I set archive mode to "delete", it works on both Linux and Windows. But for the archive mode, it does not work on Windows. 

The problem is related to appending paths in Windows. There is a method

 
{code:java}
override protected def cleanTask(entry: FileEntry): Unit{code}
in the FileStreamSource.scala file in the org.apache.spark.sql.execution.streaming package. On line 569, the !fileSystem.rename(curPath, newPath) code supposed to move source file to archive folder. However, when I debugged, I noticed that the curPath and newPath values were as follows in windows:

 
{code:java}
curPath: file:/C:/dev/be/data-integration-suite/test-data/streaming-folder/patients/patients-success.csv{code}
{code:java}
newPath: file:/C:/dev/be/data-integration-suite/archived/C:/dev/be/data-integration-suite/test-data/streaming-folder/patients/patients-success.csv{code}
It seems that absolute path of csv file were appended when creating newPath because there are two *C:/dev/be/data-integration-suite* in the newPath. This is the reason probably spark archiving does not work. Instead, newPath should be: file:/C:/dev/be/data-integration-suite/archived/test-data/streaming-folder/patients/patients-success.csv