av9ash/GitBugs huggingface dataset:
Bug: spark cannot hit oracle's index when column type is DATE
Status: Open | Priority: Minor | Resolution: 
Created: 25/Mar/24 09:01
Affects Version/s: 3.2.4

When I use spark's jdbc to pull data from oracle, it will not hit the index if the pushed filter's type in oralce is DATE.

Here is my scenario:

first I create a dataframe that reads from oracle:

val df = spark.read.format("jdbc").                                                                        
          option("url", url).                                                                          
          option("driver", driver).                                                                    
          option("user", user).                                                                   
          option("password", passwd).                                                                                                 
          option("dbtable", "select * from foobar.tbl1") 
          .load()

then I apply a filter to the dataframe like this:

df.filter("""`update_time` >= to_date('2024-03-12 06:18:17', 'yyyy-MM-dd HH:mm:ss')  """).count()

this will not hit the index on update_time column.

 

Reason：

The update_time column in Oracle is of type DATE, which is mapped to Timestamp in Spark (because the precision of DATE in Oracle is second). When I push a filter to Oracle, it triggers the following code in org.apache.spark.sql.jdbc.OracleDialect:

 

override def compileValue(value: Any): Any = value match

{     // The JDBC drivers support date literals in SQL statements written in the     // format:

{d 'yyyy-mm-dd'}

and timestamp literals in SQL statements written
    // in the format: \{ts 'yyyy-mm-dd hh:mm:ss.f...'}. For details, see
    // 'Oracle Database JDBC Developer’s Guide and Reference, 11g Release 1 (11.1)'
    // Appendix A Reference Information.
    case stringValue: String => s"'${escapeSql(stringValue)}'"
    case timestampValue: Timestamp => "\{ts '" + timestampValue + "'}"
    case dateValue: Date => "\{d '" + dateValue + "'}"
    case arrayValue: Array[Any] => arrayValue.map(compileValue).mkString(", ")
    case _ => value
  }

 

As a result, the condition "update_time >= \{ts '2024-03-12 06:18:17'} will never hit the index.

In my case, as a workaround, I changed the code to this:

{color:#cc7832}case {color}timestampValue: Timestamp =>{color:#6a8759}s"{color}{color:#6a8759}to_date({color}

{dateFormat.format(timestampValue)}

,'yyyy-MM-dd HH:mi:ss')"

 

After this modification, it worked well.
Bug: Error message regressed when creating hive table with illegal column name
Status: Open | Priority: Minor | Resolution: 
Created: 19/Mar/24 18:42
Affects Version/s: 4.0.0

The following statement used to result in:
CREATE TABLE test5(`北京, ` INT) USING HIVE;
[[INVALID_HIVE_COLUMN_NAME|https://docs.databricks.com/error-messages/error-classes.html#invalid_hive_column_name]] Cannot create the table `hive_metastore`.`srielau`.`test5` having the column `北京, ` whose name contains invalid characters ',' in Hive metastore. SQLSTATE: 42K05

Now it results in:
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe: columns has 2 elements while columns.types has 1 elements!) at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$2(HiveExternalCatalog.scala:168) at org.apache.spark.sql.hive.HiveExternalCatalog.maybeSynchronized(HiveExternalCatalog.scala:115) at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$1(HiveExternalCatalog.scala:153) at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:405) at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:391) at com.databricks.spark.util.SparkDatabricksProgressReporter$.withStatusCode(ProgressReporter.scala:34) at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:152) at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:312) at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.$anonfun$createTable$1(ExternalCatalogWithListener.scala:122) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.catalyst.MetricKeyUtils$.measure(MetricKey.scala:661) at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.$anonfun$profile$1(ExternalCatalogWithListener.scala:54) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94) at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.profile(ExternalCatalogWithListener.scala:53)
 
This may be related to:
https://github.com/apache/spark/pull/45180
Bug: Fix Resource Handling of `scalaStyleOnCompileConfig` in SparkBuild.scala
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Mar/24 03:34
Affects Version/s: 3.4.2

[https://github.com/apache/spark/blob/e01ed0da22f24204fe23143032ff39be7f4b56af/project/SparkBuild.scala#L157-L173]

 
{code:java}
val scalaStyleOnCompileConfig: String = {
    val in = "scalastyle-config.xml"
    val out = "scalastyle-on-compile.generated.xml"
    val replacements = Map(
      """customId="println" level="error"""" -> """customId="println" level="warn""""
    )
    var contents = Source.fromFile(in).getLines.mkString("\n")
    for ((k, v) <- replacements) {
      require(contents.contains(k), s"Could not rewrite '$k' in original scalastyle config.")
      contents = contents.replace(k, v)
    }
    new PrintWriter(out) {
      write(contents)
      close()
    }
    out
  } {code}
`Source.fromFile(in)` opens a `BufferedSource` resource handle, but it does not close it.
Bug: Streaming Statistics link redirect causing 302 error
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/24 22:16
Affects Version/s: 3.4.1

When using a reverse proxy, links to streaming statistics page are missing a trailing slash, which causes a redirect (302) to an incorrect path.

Essentially the same issue as https://issues.apache.org/jira/browse/SPARK-24553 but for a different link.

.../StreamingQuery/statistics?id=abcd -> .../StreamingQuery/statistics/?id=abcd

Linked PR [https://github.com/apache/spark/pull/45527/files]
Bug: Ability to only take first match for join when there are miltiple match
Status: Open | Priority: Minor | Resolution: 
Created: 14/Mar/24 03:37
Affects Version/s: 3.5.1

When joining 2 datasets, it can be desirable to only get first matching row instead of all. This can be helpful when the right table has fuplicates, and dropping duplicate is not desired due to data volume.
Bug: Spark SQL: select * from t where (false) parsed as subquery/column alias
Status: Open | Priority: Minor | Resolution: 
Created: 13/Mar/24 13:27
Affects Version/s: 3.5.0

Given the view (input4d47e3f1d26b42eabea312bd9d99ab43):

 

 
{code:java}
----------------

              o
----------------

[2B 01 0F 00 00]
----------------
{code}
 

 

the following Spark SQL query
{code:java}
select * from input4d47e3f1d26b42eabea312bd9d99ab43 where (FALSE){code}
 

outputs:

 
{code:java}
----------------

          FALSE
----------------

[2B 01 0F 00 00]
----------------
{code}
 

 

instead of an empty DataFrame.

 

A workaround is this query:
{code:java}
select o from input4d47e3f1d26b42eabea312bd9d99ab43 where true and (FALSE){code}
 

which correctly outputs:

 
{code:java}
---

  o
---
---
{code}
 

 

It seems that this comes from an ambiguity in the parser, where "where (false)" is parsed as a subquery and column alias rather than as a where clause, as can be seen by this query (projecting to column o):
{code:java}
select o from input4d47e3f1d26b42eabea312bd9d99ab43 where (FALSE){code}
 

Which outputs this error including the query plan:

 
{code:java}
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `o` cannot be resolved. Did you mean one of the following? [`FALSE`].; line 1 pos 7;
'Project ['o]
+- SubqueryAlias where
   +- Project o#1 AS FALSE#21
      +- SubqueryAlias input4d47e3f1d26b42eabea312bd9d99ab43
         +- View (`input4d47e3f1d26b42eabea312bd9d99ab43`, o#1)
            +- LogicalRDD o#1, false
{code}
 

 

From the Spark SQL grammar perspective, it can be interpreted both ways since this is an ambiguity in the grammar, but the currently implemented grammar precedence rule (parsing as a subquery alias rather than a where clause) can break the execution of automatically generated Spark SQL queries when the where clause is a simple boolean in parentheses.
Bug: Correct the ExternalSorter#writePartitionedMapOutput method comment
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Mar/24 02:51
Affects Version/s: 4.0.0

As title.

 

Return nothing when call  ExternalSorter#writePartitionedMapOutput method, so correct the function comment.

 

We update the partitions' length when call writer to close.
Bug: Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 12:50
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_325[1-9]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign names to the error classes _LEGACY_ERROR_TEMP_324[7-9]
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 12:55
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_324[7-9]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Suppress Python exceptions where PySpark is not in the Python path
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Mar/24 01:28
Affects Version/s: 4.0.0

{code}
scala> sql("create table t(i int) using json")
24/03/06 16:09:44 WARN DataSourceManager: Skipping the lookup of Python Data Sources due to the failure.
org.apache.spark.SparkException:
Error from python worker:
  /opt/homebrew/Caskroom/miniconda/base/bin/python3: Error while finding module specification for 'pyspark.daemon' (ModuleNotFoundError: No module named 'pyspark')
{code}

When PySpark is not in the Python path at all, it always shows this warning message once for every Spark session initialization.
Bug: Spark 3.5: fails to build with value defaultValueNotConstantError is not a member of object org.apache.spark.sql.errors.QueryCompilationErrors 
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 12/Jun/24 17:45
Affects Version/s: 3.5.2

PR [https://github.com/apache/spark/pull/46594] seems to have broken the Spark 3.5 build.

[ERROR] [Error] ...sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala:299: value defaultValueNotConstantError is not a member of object org.apache.spark.sql.errors.QueryCompilationErrors

I don't see that definition defined on the 3.5 branch - [https://github.com/apache/spark/blob/branch-3.5/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala]

I see it defined on master by https://issues.apache.org/jira/browse/SPARK-46905 which only went into 4.0
Bug: Casting Issue in Spark SQL: String Column Compared to Integer Value Yields Empty Results
Status: Open | Priority: Critical | Resolution: 
Created: 18/Jun/24 12:45
Affects Version/s: 3.3.2

In Spark SQL, comparing a string column to an integer value can lead to unexpected results due to type casting resulting in an empty result set.
{code:java}
case class Person(id: String, name: String)
val personDF = Seq(Person("a", "amit"), Person("b", "abhishek")).toDF()
personDF.createOrReplaceTempView("person_ddf")
val sqlQuery = "SELECT * FROM person_ddf WHERE id <> -1"
val resultDF = spark.sql(sqlQuery)
resultDF.show() // Empty result due to type casting issue 

{code}
Below is the logical and physical plan which I m getting
{code:java}
== Parsed Logical Plan ==
'Project [*]
+- 'Filter NOT ('id = -1)
   +- 'UnresolvedRelation [person_ddf], [], false

== Analyzed Logical Plan ==
id: string, name: string
Project [id#356, name#357]
+- Filter NOT (cast(id#356 as int) = -1)
   +- SubqueryAlias person_ddf
      +- View (`person_ddf`, [id#356,name#357])
         +- LocalRelation [id#356, name#357]{code}

*But when I m using the same query and table in Redshift which is based on PostGreSQL. I am getting the desired result.*


{code:java}
select * from person where id <> -1; {code}

Explain plan obtained in Redshift.


{code:java}
XN Seq Scan on person  (cost=0.00..0.03 rows=1 width=336)
  Filter: ((id)::text <> '-1'::text) {code}
 

In the execution plan for Spark, the ID column is cast as an integer, while in Redshift, the ID column is cast as a varchar.

Shouldn't Spark SQL handle this the same way as Redshift, using the datatype of the ID column rather than the datatype of -1?
Bug: Remove unnecessary installation `miniconda` from `build_and_test.yml`
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 28/Jun/24 06:07
Affects Version/s: 4.0.0

To fix GA's issue, as following:

[https://github.com/panbingkun/spark/actions/runs/9706791583/job/26791058294]

!image-2024-06-28-14-22-41-496.png|width=667,height=109!
Bug: Manage memory used by external cache
Status: Open | Priority: Critical | Resolution: 
Created: 24/Jun/24 07:03
Affects Version/s: 3.3.1

We have a scenario that use Spark together with a 3rd party file source cache, which is an independent lib and has its internal logic for cache entry creation, eviction and remove. Currently we allocate dedicated memory for this cache but the problem is that the memory can't be shared along with Spark execution/storage memory. It will be more effective for memory usage if we can count it in the _*UnifiedMemoryManager*_ and include it in memory spill logic.

We also have a requirement of memory management for a native RDD cache implementation. The existing interfaces in _*MemoryStore*_ is generally bound with Spark SerializerManager and BlockEvictionHandler. It's not easy to extend for such customized RDD cache.

For both of above requirements, we can provide a common memory register for external memory usage, beyond current RDD cache in storage memory.
Bug: [M1] Support for local variables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jun/24 16:46
Affects Version/s: 4.0.0

At the moment, variables in SQL scripts are creating session variables. We don't want this, we want variables to be considered as local (within the block/compound).

 

-To achieve this, we probably need to wait for labels support. Once we have it, we can prepend variable names with labels to make distinction between variables with the same name and only then reuse session variables mechanism to save values with such composed names.-

-If the block/compound doesn't have label, we should generate it automatically (GUID or something similar).-

 

Labels support is done, so we can use them now for local variables - they need to be able to be referenced by <labelName>.<varName>.

We cannot reuse session variables for this - we need to implement local variables per script, with scoping and shadowing.

 

Session variables need to be able to be accessed by `system.session.<varName>` at all times from the script though.

 

-Also, variables cannot recover from script failures - if a script fails, variables won’t be dropped. If we intend to reuse session variables for local variables, we should think about how to fix this.- We won't be using session variables, so this shouldn't be a problem, but need to have it in mind because wrong implementation can cause the same problem.

 

Use [Session Variables PR|https://github.com/apache/spark/pull/40474/files] as a reference in which places variable changes need to happen.
Bug: Support dict type for pyspark.sql.functions.lit
Status: Open | Priority: Major | Resolution: 
Created: 19/Jun/24 14:38
Affects Version/s: 4.0.0

Currently, the `lit` function in pyspark doesn't accept dict in order to create maps
{noformat}
F.lit({"a":1})

pyspark.errors.exceptions.captured.SparkRuntimeException: [UNSUPPORTED_FEATURE.LITERAL_TYPE] The feature is not supported: Literal for '{asd=2}' of class java.util.HashMap.{noformat}
Support should be added, the same way that scala has with `typedLit` to allow easier creation of maps using lit
Bug: [M0] Support for labels
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jun/24 16:40
Affects Version/s: 4.0.0

Add support for labels to SQL parser.

 

At first only BEGIN/END blocks will have labels. While adding new statements support for labels should be actively extended for that statements (WHILE,  LOOP, ...)

 

For more details:
 * Design doc in parent Jira item.
Bug: Allow setting ignoreNulls for array_agg
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/24 19:58
Affects Version/s: 4.0.0

Sparks `arrag_agg`/`collect_list` function automatically excludes `null` values. This is different from for example how postgres works
[https://www.postgresql.org/docs/17/functions-aggregate.html]

which will include nulls in their `array_agg`. 

The suggested fix is that we add an overload to `collect_list` and `collect_set` that allows configuring if nulls should be included or not.
Bug: Race condition between executor registration and heartbeat
Status: Open | Priority: Major | Resolution: 
Created: 28/Jun/24 22:01
Affects Version/s: 3.4.3

We found a race condition in our prod jobs when executor finished registration but when the it starts to heartbeat, the driver tells the executor is still unknown.

The below test case will demonstrate the issue:

test("test the executor registered states are consistent between scheduler and heartbeater") {
    val rpcEnv = sc.env.rpcEnv
    val fakeClusterManager = new FakeClusterManager(rpcEnv)
    val fakeClusterManagerRef = rpcEnv.setupEndpoint("fake-cm", fakeClusterManager)
    val listenerBus = new LiveListenerBus(sc.conf)
    val mockMetricsSystem = mock(classOf[MetricsSystem])
    listenerBus.start(sc, mockMetricsSystem)
    when(sc.listenerBus).thenReturn(listenerBus)
    when(sc.heartbeatReceiver).thenReturn(heartbeatReceiverRef)
    val fakeSchedulerBackend = new FakeSchedulerBackend(scheduler, rpcEnv, fakeClusterManagerRef)
    when(sc.schedulerBackend).thenReturn(fakeSchedulerBackend)
 
    heartbeatReceiverRef.askSync[Boolean](TaskSchedulerIsSet)
 
    // simulate the slow event listener in the listener queue before the heartbeatReceiver listener
    sc.listenerBus.addToManagementQueue(new SparkListener() {
      override def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit = {
        if (executorAdded.executorId != "driver") {
          Thread.sleep(2000)
        }
      }
    })
    // register the heartbeatReceiver listener to make sure it's after the
    // slow listener above
    sc.listenerBus.addToManagementQueue(heartbeatReceiver)
 
    fakeSchedulerBackend.start()
    val dummyExecutorEndpoint1 = new FakeExecutorEndpoint(rpcEnv)
    val dummyExecutorEndpointRef1 = rpcEnv.setupEndpoint("fake-executor-1", dummyExecutorEndpoint1)
 
    // register the executor
    fakeSchedulerBackend.driverEndpoint.askSync[Boolean](
      RegisterExecutor(executorId1, dummyExecutorEndpointRef1, "1.2.3.4", 0, Map.empty))
 
    // simulate the executor starting to heartbeat
    val response = heartbeatReceiverRef.askSync[HeartbeatResponse](
        Heartbeat(executorId1,
          Array.empty[(Long, Seq[AccumulatorV2[_, _]])],
          BlockManagerId(executorId1, "1.2.3.4", 8008),
          new ExecutorMetrics(Array.empty[Long])))
 
    assert(!response.reregisterBlockManager)
  }
 




Will proposed a PR for the fix soon.
Bug: Incompatibilities with NumPy 2.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/24 09:21
Affects Version/s: 3.5.1

PySpark references some code which was removed with NumPy 2.0.

 {{python/pyspark/pandas/strings.py}}:
 * {{np.NaN}} was removed, should be replaced with {{np.nan}}

{{python/pyspark/pandas/typedef/typehints.py}}
 * {{np.string_}} was removed, [is an alias for|https://github.com/numpy/numpy/blob/v1.26.5/numpy/__init__.pyi#L3134] {{np.bytes_}}
 * {{np.float_}} was removed, [is defined the same as|https://github.com/numpy/numpy/blob/v1.26.5/numpy/__init__.pyi#L3042-3043] {{np.double}}
 * {{np.unicode_}} was removed, [is an alias for|https://github.com/numpy/numpy/blob/v1.26.5/numpy/__init__.pyi#L3148] {{np.str_}}
Bug: On-demand shuffle migration peer refresh during decommission
Status: Open | Priority: Major | Resolution: 
Created: 16/Jun/24 12:43
Affects Version/s: 3.1.3

Currently the shuffle migration peers is refreshed every 30s by default. It could be more effecient if we refresh the peer immediately once there is a peer aborted.

(Strictly speaking, we only wait 30s to refresh the new peer when the there no queued peers (i.e., `ShuffleMigrationRunnable`) in the `shuffleMigrationPool`.)