av9ash/GitBugs huggingface dataset:
Bug: Set file.encoding to UTF-8 for SBT tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/22 10:15
Affects Version/s: 3.3.0

{code}
$ LANG=C.UTF-8 build/sbt "sql/testOnly *.JDBCV2Suite -- -z non-ascii"
...
[info] - column name with non-ascii *** FAILED *** (2 seconds, 668 milliseconds)
[info]   "== Parsed Logical Plan ==
[info]   'Project [unresolvedalias('COUNT('?), None)]
[info]   +- 'UnresolvedRelation [h2, test, person], [], false
[info]
[info]   == Analyzed Logical Plan ==
[info]   count(?): bigint
[info]   Aggregate [count(?#x) AS count(?)#xL]
[info]   +- SubqueryAlias h2.test.person
[info]      +- RelationV2[?#x] test.person
[info]
[info]   == Optimized Logical Plan ==
[info]   Project [COUNT(U&"\540d")#xL AS count(?#x)#xL AS count(?)#xL]
[info]   +- RelationV2[COUNT(U&"\540d")#xL] test.person
[info]
[info]   == Physical Plan ==
[info]   *(1) Project [COUNT(U&"\540d")#xL AS count(?)#xL]
[info]   +- *(1) Scan org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCScan$$anon$1@df881a4 [COUNT(U&"\540d")#xL] PushedAggregates: [COUNT(`?`)], PushedFilters: [], PushedGroupByColumns: [], ReadSchema: struct<COUNT(U&"\540d"):bigint>
[info]
[info]   " did not contain "PushedAggregates: [COUNT(`名`)]" (ExplainSuite.scala:66)
{code}
Bug: [BUG] Revert: Fix taskid in the stage page task event timeline
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/22 01:20
Affects Version/s: 3.2.1

In [#32888|https://github.com/apache/spark/pull/32888], [@shahidki31|https://github.com/shahidki31] change taskInfo.index to taskInfo.taskId. However, we generally use {{index.attempt}} or {{taskId}} to distinguish tasks within a stage, not {{{}taskId.attempt{}}}.
Thus [#32888|https://github.com/apache/spark/pull/32888] was a wrong fix issue, we should revert it.
Bug: Suppress NoSuchFieldError at setMDCForTask
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/22 00:15
Affects Version/s: 3.3.0

{code}
00:57:11 2022-01-07 15:57:11.693 - stderr> Exception in thread "Executor task launch worker-0" java.lang.NoSuchFieldError: mdc
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.apache.log4j.MDCFriend.fixForJava9(MDCFriend.java:11)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.slf4j.impl.Log4jMDCAdapter.<clinit>(Log4jMDCAdapter.java:38)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.slf4j.impl.StaticMDCBinder.getMDCA(StaticMDCBinder.java:59)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.slf4j.MDC.bwCompatibleGetMDCAdapterFromBinder(MDC.java:99)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.slf4j.MDC.<clinit>(MDC.java:108)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$setMDCForTask(Executor.scala:750)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:441)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at java.base/java.lang.Thread.run(Thread.java:833)
{code}
Bug: Push down filters through RebalancePartitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 10:11
Affects Version/s: 3.3.0

How to reproduce this issue:

{code:scala}
spark.sql("SELECT * FROM (SELECT /*+ REBALANCE */  * from range(10)) t11 WHERE id = 3").explain(true)
{code}
Output:
{noformat}
== Optimized Logical Plan ==
Filter (id#0L = 3)
+- RebalancePartitions
   +- Range (0, 10, step=1, splits=None)
{noformat}

Expected:
{noformat}
== Optimized Logical Plan ==
RebalancePartitions
+- Filter (id#0L = 3)
   +- Range (0, 10, step=1, splits=None)

{noformat}
Bug: Upgrade scalatestplus artifacts to 3.3.0-SNAP3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/22 06:49
Affects Version/s: 3.3.0

It always throws exception:
{noformat}
[error] /Users/yumwang/spark/core/src/test/scala/org/apache/spark/SparkContextInfoSuite.scala:22:8: Symbol 'type org.scalactic.TripleEquals' is missing from the classpath.
[error] This symbol is required by 'trait org.scalatest.Assertions'.
[error] Make sure that type TripleEquals is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
[error] A full rebuild may help if 'Assertions.class' was compiled against an incompatible version of org.scalactic.
[error] import org.scalatest.Assertions
[error]        ^
[error] one error found
[error] (core / Test / compileIncremental) Compilation failed
{noformat}

How to reproduce:

{code:sh}
 SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStability*Suite"
{code}
Bug: Replace ApacheCommonBase64 with JavaBase64 for string fucntions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 02:52
Affects Version/s: 3.3.0

Replace dependency on third-party libraries with native support(https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html
) for Base-64 encode/decode.


1. Performace gain

http://java-performance.info/base64-encoding-and-decoding-performance/

2. reduce dependencies afterward
Bug: Orc struct serializer should look up field converters in an array rather than a linked list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 23:07
Affects Version/s: 3.3.0

The OrcSerializer's struct converter uses an index to look up a field converter in a linked list, resulting in a n*(n/2) average complexity per row (where n is the field count).

Simply converting the linked list to an array brings performance gains, especially for wide structs.
Bug: SQL function `split` should return an array of non-nullable elements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 03:34
Affects Version/s: 3.3.0

Currently, {{split}} [returns the data type|https://github.com/apache/spark/blob/08dd010860cc176a33073928f4c0780d0ee98a08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala#L532] {{ArrayType(StringType)}} which means the resultant array can contain nullable elements. However I do not see any case where the array can contain nulls.

In the case where either the provided string or delimiter is NULL, the output will be a NULL array. In case of empty string or no chars between delemiters, the output array will contain empty strings but never NULLs. So I propose we change the return type of {{split}} to mark elements as non-null.
Bug: An outer-join using joinWith on DataFrames returns Rows with null fields instead of null values
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 06/Jan/22 16:04
Affects Version/s: 3.0.0

Doing an outer-join using {{joinWith}} on {{{}DataFrame{}}}s used to return missing values as {{null}} in Spark 2.4.8, but returns them as {{Rows}} with {{null}} values in Spark 3+.

The issue can be reproduced with [the following test|https://github.com/cdegroc/spark/commit/a499805bc7f40e741bfa9badec2588972e53d604] that succeeds on Spark 2.4.8 but fails starting from Spark 3.0.0.

The problem only arises when working with DataFrames: Datasets of case classes work as expected as demonstrated by [this other test|https://github.com/apache/spark/blob/v3.0.0/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala#L1200-L1223].

I couldn't find an explanation for this change in the Migration guide so I'm assuming this is a bug.

A {{git bisect}} pointed me to [that commit|https://github.com/apache/spark/commit/cd92f25be5a221e0d4618925f7bc9dfd3bb8cb59].

Reverting the commit solves the problem.

A similar solution,  but without reverting, is shown [here|https://github.com/cdegroc/spark/commit/3838dd5617dae51e8b323b07df7cb36e3e6728c3].

Happy to help if you think of another approach / can provide some guidance.
Bug: When deserializing an Orc struct, reuse the result row when possible
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/22 00:01
Affects Version/s: 3.3.0

For each Orc struct, the Orc deserializer creates a new result row object.

There are cases where the Orc deserializer can reuse the result row object, resulting in performance gains.
Bug: Fix flaky test SPARK-37578
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jan/22 07:09
Affects Version/s: 3.2.0

2022-01-22T01:58:29.8444339Z [0m[[0m[0minfo[0m] [0m[0m[32m- SPARK-36030: Report metrics from Datasource v2 write (90 milliseconds)[0m[0m
2022-01-22T01:58:29.9427049Z [0m[[0m[0minfo[0m] [0m[0m[31m- SPARK-37578: Update output metrics from Datasource v2 *** FAILED *** (65 milliseconds)[0m[0m
2022-01-22T01:58:29.9428038Z [0m[[0m[0minfo[0m] [0m[0m[31m  123 did not equal 246 (SQLAppStatusListenerSuite.scala:936)[0m[0m
2022-01-22T01:58:29.9428531Z [0m[[0m[0minfo[0m] [0m[0m[31m  org.scalatest.exceptions.TestFailedException:[0m[0m
2022-01-22T01:58:29.9429101Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)[0m[0m
2022-01-22T01:58:29.9429717Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)[0m[0m
2022-01-22T01:58:29.9430298Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)[0m[0m
2022-01-22T01:58:29.9430840Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)[0m[0m
2022-01-22T01:58:29.9431512Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.execution.ui.SQLAppStatusListenerSuite.$anonfun$new$61(SQLAppStatusListenerSuite.scala:936)[0m[0m
2022-01-22T01:58:29.9432305Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.execution.ui.SQLAppStatusListenerSuite.$anonfun$new$61$adapted(SQLAppStatusListenerSuite.scala:905)[0m[0m
2022-01-22T01:58:29.9432982Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:79)[0m[0m
2022-01-22T01:58:29.9433695Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:78)[0m[0m
2022-01-22T01:58:29.9434276Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.SparkFunSuite.withTempDir(SparkFunSuite.scala:221)[0m[0m
2022-01-22T01:58:29.9435040Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.execution.ui.SQLAppStatusListenerSuite.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(SQLAppStatusListenerSuite.scala:63)[0m[0m
2022-01-22T01:58:29.9435764Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:78)[0m[0m
2022-01-22T01:58:29.9436354Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:77)[0m[0m
2022-01-22T01:58:29.9437063Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.execution.ui.SQLAppStatusListenerSuite.withTempDir(SQLAppStatusListenerSuite.scala:63)[0m[0m
2022-01-22T01:58:29.9437851Z [0m[[0m[0minfo[0m] [0m[0m[31m  at org.apache.spark.sql.execution.ui.SQLAppStatusListenerSuite.$anonfun$new$60(SQLAppStatusListenerSuite.scala:905)[0m[0m
Bug: Any depth search not working in get_json_object ($..foo)
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jan/22 19:11
Affects Version/s: 3.2.0

The following example should return a value _abc_ but instead returns null
{code:java}
spark.sql("""select get_json_object('{"k":{"value":"abc"}}', '$..value') as j""").show(){code}
I checked the [spark test suite for Json expression parser | [spark/JsonExpressionsSuite.scala at f12793de205a46c75529ac20649e2e61edc0b144 · apache/spark · GitHub|https://github.com/apache/spark/blob/f12793de205a46c75529ac20649e2e61edc0b144/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala]] and the example there seems to be for a negative test case (non-existent key), so I'm not sure if this is expected.
Bug: TreeNode.argString incorrectly formats arguments of type Set[_]
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Jan/22 23:43
Affects Version/s: 3.2.0

The implementation of {{argString}} uses the following pattern for sets:

 
{code:java}
case set: Set[_] =>
  // Sort elements for deterministic behaviours
  val sortedSeq = set.toSeq.map(formatArg(_, maxFields).sorted)                                                              
  truncatedString(sortedSeq, "{", ", ", "}", maxFields) :: Nil {code}
Instead of sorting the elements of the set, the implementation sorts the characters of the strings that {{formatArg}} returns. 

The fix is simply to move the closing parenthesis to the correct location:
{code:java}
  val sortedSeq = set.toSeq.map(formatArg(_, maxFields)).sorted
{code}
Bug: Support of 'melt' function in spark
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 01/Jan/22 13:20
Affects Version/s: 3.2.0

Hello,

Un-pivoting a dataframe is currently supported in Pandas with the 'melt' function, but isn't available in spark. It's easy enough to produce this functionality from the functions module (e.g. such as the melt function in pandas-on-pyspark [here|https://github.com/apache/spark/blob/c92bd5cafe62ca5226176446735171cc877e805a/python/pyspark/pandas/frame.py#L9651]), but I was wondering whether a more native solution had been considered? It would make end-user code more lightweight at the very least; and I wonder whether it could be made more efficient than using the stack function/struct-array-explode functions.

I'm happy to try and make a PR if this is something that might be useful within spark. No worries if not, the methods above work fine.
Bug: Upgrade Arrow version < 7.0.0 for Python UDF tests in SQL
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jan/22 04:53
Affects Version/s: 3.3.0

We should better test latest PyArrow version. Now 6.0.1 is release but we're using < 5.0.0 for https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
Bug: Contribution guide is stale
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jan/22 11:02
Affects Version/s: 3.2.0

Contribution guide mentions below link to use to test on local repo before raising PR but the process has changed and documentation does not reflect it.

https://spark.apache.org/developer-tools.html#github-workflow-tests

Only digging into git log of " [.github/workflows/build_and_test.yml|https://github.com/apache/spark/commit/2974b70d1efd4b1c5cfe7e2467766f0a9a1fec82#diff-48c0ee97c53013d18d6bbae44648f7fab9af2e0bf5b0dc1ca761e18ec5c478f2]" I managed to find what the new process is. It was changed in [https://github.com/apache/spark/pull/32092] but documentation was not updated.

I am happy to contribute to fix it but apparently [https://spark.apache.org/developer-tools.html] is hosted in Apache website rather that in the Spark source code
Bug: Allow query parameters to be passed into spark.read
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jan/22 11:46
Affects Version/s: 3.2.0

This ticket is to allow query parameters to be used spark.read.

While it is possible to inject some parameters into the query using string concatenation, this doesn't work for all data types, for example binaries. In this example, the parameter rowversion is a binary which needs to be passed into the sql query. 
{code:java}
_select_sql = f'SELECT * FROM dbo.Table WHERE RowVersion > {rowversion}'

df = spark.read.format("jdbc") \
    .option("url", "jdbc:sqlserver://databaseserver.database.windows.net;databaseName=databasename") \
    .option("query", _select_sql) \
    .option("username", "sql_username") \
    .option("password", "sql_password") \
    .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
    .load() {code}
This results in the query looking this this...
{code:java}
SELECT * FROM dbo.Address WHERE RowVersion > bytearray(b'\x00\x00\x00\x00\x02\xdf=\xf5') {code}
As far as I know, there is no way to do this currently.
Bug: Mark legacy file naming functions as deprecated in FileCommitProtocol
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jan/22 06:23
Affects Version/s: 3.3.0

[FileCommitProtocol|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala] is the class to commit Spark job output (staging file & directory renaming, etc). During Spark 3.2 development, we added new functions into this class to allow more flexible output file naming (the PR detail is [here|https://github.com/apache/spark/pull/33012]). We didn’t delete the existing file naming functions (newTaskTempFile(ext) & newTaskTempFileAbsPath(ext)), because we were aware of many other downstream projects or codebases already implemented their own custom implementation for FileCommitProtocol. Delete the existing functions would be a breaking change for them when upgrading Spark version, and we would like to avoid this unpleasant surprise for anyone if possible. But we also need to clean up legacy as we evolve our codebase.

So for next step, I would like to propose:
 * Spark 3.3 (now): Add @deprecate annotation to legacy functions in FileCommitProtocol - [newTaskTempFile(ext)|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala#L98] & [newTaskTempFileAbsPath(ext)|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala#L135].
 * Next Spark major release (or whenever people feel comfortable): delete the legacy functions mentioned above from our codebase.
Bug: read_excel's parameter - mangle_dupe_cols is used to handle duplicate columns but fails if the duplicate columns are case sensitive.
Status: In Progress | Priority: Minor | Resolution: 
Created: 24/Jan/22 22:52
Affects Version/s: 3.2.0

mangle_dupe_cols - default is True
So ideally it should have handled duplicate columns, but in case the columns are case sensitive it fails as below.

AnalysisException: Reference '{{{}Sheet.col{}}}' is ambiguous, could be Sheet.col, Sheet.col.

Where two columns are Col and cOL

In the best practices, there is a mention of not to use case sensitive columns - [https://koalas.readthedocs.io/en/latest/user_guide/best_practices.html#do-not-use-duplicated-column-names]

Either the docs for read_excel/mangle_dupe_cols have to be updated about this or it has to be handled.
Bug: Fix the API doc for window to say it supports TimestampNTZType too as timeColumn
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jan/22 06:57
Affects Version/s: 3.2.0

window function supports not only TimestampType but also TimestampNTZType but the API docs doesn't mention TimestampNTZType.

This issue is similar to SPARK-38016, but this issue affects 3.2.0 too, so I separate the tickets.
Bug: Cannot use <func>_outer generators in a lateral view
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jan/22 20:54
Affects Version/s: 3.3.0

This works:
{noformat}
select * from values 1, 2 lateral view outer explode(array()) as b;
{noformat}
But this does not work:
{noformat}
select * from values 1, 2 lateral view explode_outer(array()) as b;
{noformat}
It produces the error:
{noformat}
Error in query: Column 'b' does not exist. Did you mean one of the following? [col1]; line 1 pos 26;
{noformat}
Similarly, this works:
{noformat}
select * from values 1, 2
lateral view outer inline(array(struct(1, 2, 3))) as b, c, d;
{noformat}
But this does not:
{noformat}
select * from values 1, 2
lateral view inline_outer(array(struct(1, 2, 3))) as b, c, d;
{noformat}
It produces the error:
{noformat}
Error in query: Column 'b' does not exist. Did you mean one of the following? [col1]; line 2 pos 0;
{noformat}
Bug: Error while joining two tables with non-english field names
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/22 09:53
Affects Version/s: 3.2.0

While trying to join two tables with non-english field names in postgresql with query like
"select view1.`Имя1` , view1.`Имя2`, view2.`Имя3` from view1 left join  view2 on view1.`Имя2`=view2.`Имя4`"
we get an error which says that there is no field "`Имя4`" (field name is surrounded by backticks).
It appears that to get the data from the second table it constructs query like
SELECT "Имя3","Имя4" FROM "public"."tab2"  WHERE ("`Имя4`" IS NOT NULL) 
and these backticks are redundant in WHERE clause.
Bug: Use `rbac.authorization.k8s.io/v1` instead of `v1beta1`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jan/22 13:45
Affects Version/s: 3.3.0

*$ kubectl version*
Client Version: version.Info\{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info\{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

*$ k apply -f  resource-managers/kubernetes/integration-tests/dev/spark-rbac.yaml*
namespace/spark created
serviceaccount/spark-sa created
*unable to recognize "resource-managers/kubernetes/integration-tests/dev/spark-rbac.yaml": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1beta1"*
*unable to recognize "resource-managers/kubernetes/integration-tests/dev/spark-rbac.yaml": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"*
Bug: Inconsistent behavior from JSON option allowNonNumericNumbers
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jan/22 17:20
Affects Version/s: 3.2.0

The behavior of the JSON option allowNonNumericNumbers is not consistent:

1. Some NaN and Infinity values are still parsed when the option is set to false

2. Some values are parsed differently depending on whether they are quoted or not (see results for positive and negative Infinity)
h2. Input data
{code:java}
{ "number": "NaN" }
{ "number": NaN }
{ "number": "+INF" }
{ "number": +INF }
{ "number": "-INF" }
{ "number": -INF }
{ "number": "INF" }
{ "number": INF }
{ "number": Infinity }
{ "number": +Infinity }
{ "number": -Infinity }
{ "number": "Infinity" }
{ "number": "+Infinity" }
{ "number": "-Infinity" }
{code}
h2. Setup
{code:java}
import org.apache.spark.sql.types._

val schema = StructType(Seq(StructField("number", DataTypes.FloatType, false))) {code}
h2. allowNonNumericNumbers = false
{code:java}
spark.read.format("json").schema(schema).option("allowNonNumericNumbers", "false").json("nan_valid.json")

df.show

+---------+
|   number|
+---------+
|      NaN|
|     null|
|     null|
|     null|
|     null|
|     null|
|     null|
|     null|
|     null|
|     null|
|     null|
| Infinity|
|     null|
|-Infinity|
+---------+ {code}
h2. allowNonNumericNumbers = true
{code:java}
val df = spark.read.format("json").schema(schema).option("allowNonNumericNumbers", "true").json("nan_valid.json") 

df.show

+---------+
|   number|
+---------+
|      NaN|
|      NaN|
|     null|
| Infinity|
|     null|
|-Infinity|
|     null|
|     null|
| Infinity|
| Infinity|
|-Infinity|
| Infinity|
|     null|
|-Infinity|
+---------+{code}
Bug: Improve test suite ExternalCatalogSuite
Status: In Progress | Priority: Minor | Resolution: 
Created: 25/Jan/22 17:14
Affects Version/s: 3.3.0

Test suite *ExternalCatalogSuite.scala* can be optimized by removing repetitive code by replacing them with already available utility function with some minor changes. This will reduce redundant code, simplify the suite and improve readability.