av9ash/GitBugs huggingface dataset:
Bug: Collated strings in complex types supporting operations reverse, array_join, concat, map
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/24 08:10
Affects Version/s: 4.0.0

Add proper support for complex types containing collated strings in operations reverse, array_join, concat, map (create). Examples:
{code:java}
select reverse('abc' collate utf8_binary_lcase);
select reverse(array('a' collate utf8_binary_lcase, 'b' collate utf8_binary_lcase));
select array_join(array('a' collate utf8_binary_lcase, 'b' collate utf8_binary_lcase), ', ' collate utf8_binary_lcase);
select concat('a' collate utf8_binary_lcase, 'b' collate utf8_binary_lcase);
select map('a' collate utf8_binary_lcase, 1, 'A' collate utf8_binary_lcase, 2);{code}
Bug: Add TPC-DS testing infrastructure for collations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/24 15:33
Affects Version/s: 4.0.0

As collation support grows across all SQL features and new collation types are added, we need to have reliable testing model covering as many standard SQL capabilities as possible.

We can utilize TPC-DS testing infrastructure already present in Spark. The idea is to vary TPC-DS table string columns by adding multiple collations with different ordering rules and case sensitivity, producing new tables. These tables should yield the same results against predefined TPC-DS queries for certain batches of collations. For example, when comparing query runs on table where columns are first collated as UTF8_BINARY and then as UTF8_BINARY_LCASE, we should be getting same results after converting to lowercase.

Introduce new query suite which tests the described behavior with available collations (utf8_binary and unicode) combined with case conversions (lowercase, uppercase, randomized case for fuzzy testing).
Bug: Resolve AbstractDataType simpleStrings for StringTypeCollated
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Mar/24 13:18
Affects Version/s: 4.0.0

*SPARK-47296* introduced a change to fail all unsupported functions. Because of this change expected *inputTypes* in *ExpectsInputTypes* had to be changed. This change introduced a change on user side which will print *"STRING_ANY_COLLATION"* in places where before we printed *"STRING"* when an error occurred. Concretely if we get an input of Int where *StringTypeAnyCollation* was expected, we will throw this faulty message for users.
Bug: IN operator support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/24 10:37
Affects Version/s: 4.0.0

At this point following query works fine:
```
 sql("select * from t1 where ucs_basic_lcase in ('aaa' collate 'ucs_basic_lcase', 'bbb' collate 'ucs_basic_lcase')").show()

```

But if we were to miss explicit collate or even mix collations:
```
      sql("select * from t1 where ucs_basic_lcase in ('aaa' collate 'ucs_basic_lcase', 'bbb'").show()

```

Query would still run and return invalid results.
Bug: Allow extensions to log extended information in explain plan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/24 18:31
Affects Version/s: 4.0.0

With session extensions, Spark planning can be extended to apply additional rules and modify the execution plan. If an extension replaces a node in the plan, the new node will be displayed in the plan. However, it is sometimes useful for extensions provided extended information to the end user to explain the impact of the extension. For instance an extension may automatically enable/disable some feature that it provides and can provide this extended information in the plan. 
The proposal is to optionally turn on extended plan information from extensions. Extensions can add additional planning information via a new interface that internally uses a new TreeNodeTag, say 'explainPlan'.
Bug: Canonicalize With expressions by re-assigning IDs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Mar/24 18:36
Affects Version/s: 4.0.0

The current canonicalization of `With` expressions takes into account the ID of the common expressions, which comes from a global monotonically increasing ID. This means that queries with `With` expressions (e.g. `NULLIF` expressions) will have inconsistent canonicalizations.
Bug: Introduce Structured Logging Framework
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/24 18:47
Affects Version/s: 4.0.0

Introduce Structured Logging Framework as per [https://docs.google.com/document/d/1rATVGmFLNVLmtxSpWrEceYm7d-ocgu8ofhryVs4g3XU/edit?usp=sharing] .
 * The default logging output format will be json lines. For example 
{code:java}
{
   "ts":"2023-03-12T12:02:46.661-0700",
   "level":"ERROR",
   "msg":"Cannot determine whether executor 289 is alive or not",
   "context":{
       "executor_id":"289"
   },
   "exception":{
      "class":"org.apache.spark.SparkException",
      "msg":"Exception thrown in awaitResult",
      "stackTrace":"..."
   },
   "source":"BlockManagerMasterEndpoint"
} {code}

 * Introduce a new configuration `spark.log.structuredLogging.enabled` to control the default log4j configuration. Users can set it as false to get plain text log outputs
 * The change will start with logError method. Example changes on the API: from
`logError(s"Cannot determine whether executor $executorId is alive or not.", e)` to `logError(log"Cannot determine whether executor ${MDC(EXECUTOR_ID, executorId)} is alive or not.", e)`
Bug: Inferring `dict` as `MapType` from Pandas DataFrame to allow DataFrame creation.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/24 13:21
Affects Version/s: 4.0.0

Currently the PyArrow infers the Pandas dictionary field as StructType instead of MapType, so Spark can't handle the schema properly:
{code:java}
>>> pdf = pd.DataFrame({"str_col": ['second'], "dict_col": [{'first': 0.7, 'second': 0.3}]})
>>> pa.Schema.from_pandas(pdf)
str_col: string
dict_col: struct<first: double, second: double>
  child 0, first: double
  child 1, second: double
{code}
We cannot handle this case since we use PyArrow for schema creation.
Bug: Fix race condition between maintenance thread and task thead for RocksDB snapshot
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/24 14:58
Affects Version/s: 3.5.0

There are currently some race conditions between maintenance thread and task thread which can result in corrupted checkpoint state.
 # The maintenance thread currently relies on class variable {{lastSnapshot}} to find the latest checkpoint and uploads it to DFS. This checkpoint can be modified at commit time by Task thread if a new snapshot is created.
 # The task thread does not reset lastSnapshot at load time, which can result in newer snapshots (if a old version is loaded) being considered valid and uploaded to DFS. This results in VersionIdMismatch errors.

This issue proposes to fix these issues by guarding latestSnapshot variable modification, and setting latestSnapshot properly at load time.
Bug: PySpark workers dying in daemon mode idle queue fail query
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/24 12:28
Affects Version/s: 3.3.4

PySpark workers may die after entering the idle queue in `PythonWorkerFactory`. This may happen because of code that runs in the process, or external factors.

When drawn from the warmpool, such a worker will result in an I/O exception on the first read/write .
Bug: Validate the spark connect session on server side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Mar/24 11:53
Affects Version/s: 3.5.0

Currently SparkConnect only validates that the server-side session is the same on the client side, during the receipt of the response from the server. This allow that the sparkconnect server executes the command on the wrong spark session, which can succeed, and the client will bail-out only when it get the response back.
Bug: Fix Postgres bit array issue 'Cannot cast to boolean'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 10:19
Affects Version/s: 4.0.0

{code:java}
[info]   Cause: org.postgresql.util.PSQLException: Cannot cast to boolean: "10101"
[info]   at org.postgresql.jdbc.BooleanTypeUtil.cannotCoerceException(BooleanTypeUtil.java:99)
[info]   at org.postgresql.jdbc.BooleanTypeUtil.fromString(BooleanTypeUtil.java:67)
[info]   at org.postgresql.jdbc.ArrayDecoding$7.parseValue(ArrayDecoding.java:267)
[info]   at org.postgresql.jdbc.ArrayDecoding$AbstractObjectStringArrayDecoder.populateFromString(ArrayDecoding.java:128)
[info]   at org.postgresql.jdbc.ArrayDecoding.readStringArray(ArrayDecoding.java:763)
[info]   at org.postgresql.jdbc.PgArray.buildArray(PgArray.java:320)
[info]   at org.postgresql.jdbc.PgArray.getArrayImpl(PgArray.java:179)
[info]   at org.postgresql.jdbc.PgArray.getArray(PgArray.java:116)
[info]   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeGetter$25(JdbcUtils.scala:548)
[info]   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.nullSafeConvert(JdbcUtils.scala:561)
[info]   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeGetter$24(JdbcUtils.scala:548)
[info]   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeGetter$24$adapted(JdbcUtils.scala:545)
[info]   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:365)
[info]   at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:346)
[info]   at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
[info]   at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) {code}
Bug: Structured logging framework: support log concatenation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/24 22:51
Affects Version/s: 4.0.0

Allow log syntax like
{code:java}
log"${MDC(CONFIG, SHUFFLE_MAPOUTPUT_MIN_SIZE_FOR_BROADCAST.key)} " +
  log"(${MDC(MIN_SIZE, minSizeForBroadcast.toString)} bytes) " +
  log"must be <= spark.rpc.message.maxSize (${MDC(MAX_SIZE, maxRpcMessageSize.toString)} " +
  log"bytes) to prevent sending an rpc message that is too large." {code}
Bug: SubstringIndex, StringLocate (all collations)
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/24 06:37
Affects Version/s: 4.0.0

Enable collation support for the *SubstringIndex* and *StringLocate* built-in string functions in Spark. First confirm what is the expected behaviour for these functions when given collated strings, and then move on to implementation and testing. One way to go about this is to consider using {_}StringSearch{_}, an efficient ICU service for string matching. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *SubstringIndex* and *StringLocate* functions so that they support all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class, as well as _StringSearch_ using the [ICU user guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html] and [ICU docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html]. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Legacy support for map normalization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 21:38
Affects Version/s: 4.0.0

Add legacy support for creating a map without normalizing keys before inserting in `ArrayBasedMapBuilder`.

 

Key normalization change can be found in this PR: https://issues.apache.org/jira/browse/SPARK-47563
Bug: Memory leak in StructType column's .dropFields method
Status: Open | Priority: Major | Resolution: 
Created: 31/Mar/24 10:10
Affects Version/s: 3.5.1

We noticed that when we're using the .dropFields method, for example by calling

df.withColum(column, col(column).dropFields("fieldToDrop")) the heap "explodes". we noticed it when our integration tests, for which 512MB heap is always enough started crashing with OutOfMemoryException (even when we increased the heap size to 8GB) and after a lot of trial and error we found that when we remove the .dropFields method the heap goes back down
Bug: Incorrect to calculate the concurrent task number
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/24 06:41
Affects Version/s: 4.0.0

The below test case failed,

 
{code:java}
test("problem of calculating the maximum concurrent task") {
  withTempDir { dir =>
    val discoveryScript = createTempScriptWithExpectedOutput(
      dir, "gpuDiscoveryScript", """{"name": "gpu","addresses":["0", "1", "2", "3"]}""")

    val conf = new SparkConf()
      // Setup a local cluster which would only has one executor with 2 CPUs and 1 GPU.
      .setMaster("local-cluster[1, 6, 1024]")
      .setAppName("test-cluster")
      .set(WORKER_GPU_ID.amountConf, "4")
      .set(WORKER_GPU_ID.discoveryScriptConf, discoveryScript)
      .set(EXECUTOR_GPU_ID.amountConf, "4")
      .set(TASK_GPU_ID.amountConf, "2")
      // disable barrier stage retry to fail the application as soon as possible
      .set(BARRIER_MAX_CONCURRENT_TASKS_CHECK_MAX_FAILURES, 1)
    sc = new SparkContext(conf)
    TestUtils.waitUntilExecutorsUp(sc, 1, 60000)

    // Setup a barrier stage which contains 2 tasks and each task requires 1 CPU and 1 GPU.
    // Therefore, the total resources requirement (2 CPUs and 2 GPUs) of this barrier stage
    // can not be satisfied since the cluster only has 2 CPUs and 1 GPU in total.
    assert(sc.parallelize(Range(1, 10), 2)
      .barrier()
      .mapPartitions { iter => iter }
      .collect() sameElements Range(1, 10).toArray[Int])
  }
} {code}
The error log

 
 
[SPARK-24819]: Barrier execution mode does not allow run a barrier stage that requires more slots than the total number of slots in the cluster currently. Please init a new cluster with more resources(e.g. CPU, GPU) or repartition the input RDD(s) to reduce the number of slots required to run this barrier stage.
org.apache.spark.scheduler.BarrierJobSlotsNumberCheckFailed: [SPARK-24819]: Barrier execution mode does not allow run a barrier stage that requires more slots than the total number of slots in the cluster currently. Please init a new cluster with more resources(e.g. CPU, GPU) or repartition the input RDD(s) to reduce the number of slots required to run this barrier stage.
at org.apache.spark.errors.SparkCoreErrors$.numPartitionsGreaterThanMaxNumConcurrentTasksError(SparkCoreErrors.scala:241)
at org.apache.spark.scheduler.DAGScheduler.checkBarrierStageWithNumSlots(DAGScheduler.scala:576)
at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:654)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:1321)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3055)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3046)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3035)
Bug: Using from_json with columnNameOfCorruptRecord and ArrayType schema not working
Status: Open | Priority: Major | Resolution: 
Created: 31/Mar/24 10:19
Affects Version/s: 3.5.1

When using schemas of type ArrayType using the columnNameOfCorruptRecord parameter is not changing anything... even though the schema does contain the field it is not populated during parsing

The cause is line 574 in JsonExpressions.scala and onward in running FailureSafeParser where only when the schema is StructType this field is actually handled and taken into account
Bug: Use errorCapturingIdentifier rule in more places to improve error messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 23:15
Affects Version/s: 4.0.0

errorCapturingIdentifier parses identifier with included '-' to raise INVALID_IDENTIFIER 

instead of SYNTAX_ERROR for non-delimited identifiers containing a hyphen.
It is meant to be used wherever the context is not that of an expression
This Jira replaces a few missed identifiers with that rule.
Bug: LATERAL VIEW EXPLODE(exp) unnecessarily calculates the exp twice due to InferFiltersFromGenerate
Status: Open | Priority: Major | Resolution: 
Created: 24/Mar/24 17:07
Affects Version/s: 3.5.0

We have a query like this:

{{WITH staging AS (}}
{{  SELECT}}
{{    col1,}}
{{    FROM_JSON(MY_PYTHON_UDF(col2), 'map<string, string>') AS mapping}}
{{  FROM my_table}}
{{)}}
{{SELECT col1, key1, value1}}
{{FROM staging}}
{{{}LATERAL VIEW EXPLODE({}}}{{{}mapping{}}}{{{}) AS key1, value1{}}}

 

*Current behavior:*

The generated plan uses two invocations of MY_PYTHON_UDF:

{{+- EvalExternalUDF}}

{{  +- Project (get rid of mapping)}}

{{    +- Filter (size(mapping) > 0 && isnotnull(mapping))}}

{{      +- EvalExternalUDF }}

However, if we use {{{}LATERAL VIEW OUTER EXPLODE{}}}, then there is only one {{{}EvalExternalUDF{}}}, since there is no need to do the filter in between.

 

*Expected behavior:*  Even with {{LATERAL VIEW EXPLODE, there is no need to do project and then EvalExternalUDF again.  There should be just one invocation of EvalExternalUDF.}}
Bug: Support subquery correlation joining on map attributes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/24 19:07
Affects Version/s: 3.5.0

Currently, when a subquery is correlated on a condition like `outer_map[1] = inner_map[1]`, DecorrelateInnerQuery generates a join on the map itself,
which is unsupported, so the query cannot run - for example:
 
{code:java}
scala> Seq(Map(0 -> 0)).toDF.createOrReplaceTempView("v")scala> sql("select v1.value[0] from v v1 where v1.value[0] > (select avg(v2.value[0]) from v v2 where v1.value[1] = v2.value[1])").explain
org.apache.spark.sql.AnalysisException: [UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.UNSUPPORTED_CORRELATED_REFERENCE_DATA_TYPE] Unsupported subquery expression: Correlated column reference 'v1.value' cannot be map type. SQLSTATE: 0A000; line 1 pos 49
at org.apache.spark.sql.errors.QueryCompilationErrors$.unsupportedCorrelatedReferenceDataTypeError(QueryCompilationErrors.scala:2463)
... {code}

However, if we rewrite the query to pull out the map access `outer_map[1]` into the outer plan, it succeeds:
 
{code:java}
scala> sql("""with tmp as (
select value[0] as value0, value[1] as value1 from v
)
select v1.value0 from tmp v1 where v1.value0 > (select avg(v2.value0) from tmp v2 where v1.value1 = v2.value1)""").explain{code}
Another point that can be improved is that, even if the data type supports join, we still don’t need to join on the full attribute, and we can get a better plan by doing the same rewrite to pull out the extract expression.
Bug: Support `spark.kubernetes.jars.avoidDownloadSchemes` for K8s Cluster Mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/24 06:25
Affects Version/s: 4.0.0

Under K8s cluster deployment mode, all the jars, including primary resource jar, jars from {{--jars}} or {{spark.jars}}, will be downloaded to driver local and then served to executors through file server running on driver.

When jars are big and the application requests a lot of executors, the massive concurrent jars download from the driver will cause network saturation. In this case, the executors jar download will timeout, causing executors to be terminated. From user point of view, the application is trapped in the loop of massive executor loss and re-provision, but never gets enough live executors as requested, which leads to job SLA breach or sometimes job failure.
Bug: Spark creates lot of tiny blocks for a single driverLog file of size less than a dfs.blocksize
Status: Open | Priority: Major | Resolution: 
Created: 28/Mar/24 06:42
Affects Version/s: 3.3.2

Upon reviewing the spark code, found that /user/spark/driverLogs are synced to HDFS with hsync option as shown below.
{code:java}
 hdfsStream.hsync(EnumSet.allOf(classOf[HdfsDataOutputStream.SyncFlag]))

Ref: https://github.com/apache/spark/blob/a3c04ec1145662e4227d57cd953bffce96b8aad7/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala{code}
As a result of this we see lot of tiny blocks getting synced every 5 seconds with a new block. So we see a small HDFS file with 8 blocks as shown in the below example.
{code:java}
[root@ccycloud-3.smajeti.root.comops.site subdir0]# hdfs fsck /user/spark/driverLogs/application_1710495774861_0002_driver.log
Connecting to namenode via https://ccycloud-3.smajeti.root.comops.site:20102/fsck?ugi=hdfs&path=%2Fuser%2Fspark%2FdriverLogs%2Fapplication_1710495774861_0002_driver.log
FSCK started by hdfs (auth:KERBEROS_SSL) from /10.140.136.139 for path /user/spark/driverLogs/application_1710495774861_0002_driver.log at Thu Mar 28 06:37:29 UTC 2024

Status: HEALTHY
 Number of data-nodes:	4
 Number of racks:		1
 Total dirs:			0
 Total symlinks:		0

Replicated Blocks:
 Total size:	157574 B
 Total files:	1
 Total blocks (validated):	8 (avg. block size 19696 B)
 Minimally replicated blocks:	8 (100.0 %) {code}
HdfsDataOutputStream.SyncFlag includes two flags UPDATE_LENGTH and END_BLOCK. This has been an expected behavior for some time now and these flags will help visualize the latest size of the HDFS Driver log file and to achieve that, blocks are being ended/closed every 5-second sync. Every new sync will create a new block for the same HDFS driver log file. This hysnc behavior was started after fixing SPARK-29105 (SHS may delete driver log file of in-progress application) 5 years back.

But this leaves Namenode to manage a lot of meta and becomes an overhead at times in large clusters.
{code:java}
public static enum SyncFlag {
        UPDATE_LENGTH,
        END_BLOCK;

        private SyncFlag() {
        }
    }
{code}
I don't see any configurable option to avoid this and avoiding this type of hsync may have some side effects in spark as we saw SPARK-29105 bug.
We only have two options that needs manual intevention
1. Keep cleaning these Driver logs after some time
2. Keep merging these small block files into files with 128MB 

Can we provide some customizable option to merge these blocks while closing the spark-shell or during closing the driver log file?
Bug: Initial State without state reader implementation for State API v2.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/24 17:13
Affects Version/s: 4.0.0

This PR adds support for users to provide a Dataframe that can be used to instantiate state for the query in the first batch for arbitrary state API v2.

Note that populating the initial state will only happen for the first batch of the new streaming query. Trying to re-initialize state for the same grouping key will result in an error.