av9ash/GitBugs huggingface dataset:
Bug: RocksDB state store: Add additional metrics for better observability into state store operations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/21 05:18
Affects Version/s: 3.1.2

Proposing adding following new metrics to {{customMetrics}} under the {{stateOperators}} in {{StreamingQueryProgress}} event These metrics help have better visibility into the RocksDB based state store in streaming jobs.
 * {{rocksdbGetCount}} number of get calls to the DB (doesn’t include Gets from WriteBatch - in memory batch used for staging writes) 
 * {{rocksdbPutCount}} number of put calls to the DB (doesn’t include Puts to WriteBatch - in memory batch used for staging writes)
 * {{rocksdbTotalBytesReadByGet/rocksdbTotalBytesWrittenByPut}}: Number of uncompressed bytes read/written by get/put operations
 * {{rocksdbReadBlockCacheHitCount/rocksdbReadBlockCacheMissCount}} indicates how much of the block cache in RocksDB is useful or not and avoiding local disk reads
 * {{rocksdbTotalBytesReadByCompaction/rocksdbTotalBytesWrittenByCompaction}}: How many bytes the compaction process read from disk and written to disk. 
 * {{rocksdbTotalCompactionTime}}: Time (in ns) took for compactions (both background and the optional compaction initiated during the commit)
 * {{rocksdbWriterStallDuration}} Time (in ns) the writer has stalled due to a background compaction or flushing of the immutable memtables to disk. 
 * {{rocksdbTotalBytesReadThroughIterator}} Some of the stateful operations (such as timeout processing in FlatMapGroupsWithState and watermarking) requires reading entire data in DB through iterator. This metric tells the total size of uncompressed data read using the iterator.
Bug: Use atomic renames for diskstore block writes & do "fall-through" reads
Status: Open | Priority: Major | Resolution: 
Created: 28/Jul/21 19:09
Affects Version/s: 3.3.0

This is a potential follow-up for SPARK-36058. Since we could end up in the situation where we have a partial write we can't do fall through reads to the diskStore. A solution to this is to use atomic renames on successful write of a block which would make it safe to assume that any on-disk representation of a block represented the full block.
Bug: Adjust `astype` of fractional Series with missing values to follow pandas
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/21 23:50
Affects Version/s: 3.2.0

{code:java}
>>> pser = pd.Series([1, 2, np.nan], dtype=float)
>>> psser = ps.from_pandas(pser)
>>> pser.astype(int)
...
 ValueError: Cannot convert non-finite values (NA or inf) to integer
>>> psser.astype(int)
 0 1.0
 1 2.0
 2 NaN
 dtype: float64
{code}
As shown above, astype of Series with fractional missing values doesn't behave the same as pandas, we ought to adjust that.
Bug: conv() inconsistently handles invalid strings with > 64 invalid characters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 19:23
Affects Version/s: 3.2.0

1/ SPARK-33428 fixed ArrayIndexOutofBoundsException but introduced a new inconsistency in behaviour where the returned value is different above the 64 char threshold.

 
{noformat}
scala> spark.sql("select conv(repeat('?', 64), 10, 16)").show
+---------------------------+
|conv(repeat(?, 64), 10, 16)|
+---------------------------+
|                          0|
+---------------------------+




scala> spark.sql("select conv(repeat('?', 65), 10, 16)").show
+---------------------------+
|conv(repeat(?, 65), 10, 16)|
+---------------------------+
|           FFFFFFFFFFFFFFFF|
+---------------------------+




scala> spark.sql("select conv(repeat('?', 65), 10, -16)").show
+----------------------------+
|conv(repeat(?, 65), 10, -16)|
+----------------------------+
|                          -1|
+----------------------------+




scala> spark.sql("select conv(repeat('?', 64), 10, -16)").show
+----------------------------+
|conv(repeat(?, 64), 10, -16)|
+----------------------------+
|                           0|
+----------------------------+{noformat}
 

2/ conv should return result equal to max unsigned long value in base toBase when there is overflow
{code:java}
scala> spark.sql(select conv('aaaaaaa0aaaaaaa0a', 16, 10)).show 
// which should be 18446744073709551615

+-------------------------------+
|conv(aaaaaaa0aaaaaaa0a, 16, 10)|
+-------------------------------+
|           12297828695278266890|
+-------------------------------+
{code}
Bug: Group exception messages in core/rdd 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 18:13
Affects Version/s: 3.3.0

'core/src/main/scala/org/apache/spark/rdd'
|| Filename                        ||   Count ||
| BlockRDD.scala                  |       2 |
| DoubleRDDFunctions.scala        |       1 |
| EmptyRDD.scala                  |       1 |
| HadoopRDD.scala                 |       1 |
| LocalCheckpointRDD.scala        |       1 |
| NewHadoopRDD.scala              |       1 |
| PairRDDFunctions.scala          |       7 |
| PipedRDD.scala                  |       1 |
| RDD.scala                       |       8 |
| ReliableCheckpointRDD.scala     |       4 |
| ReliableRDDCheckpointData.scala |       1 |
Bug: ResolveAggregateFunctions should work with nested fields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/21 17:51
Affects Version/s: 3.2.0

A sort after Aggregate can fail to resolve if it contains nested fields. For example

{code:java}
SELECT c.x, SUM(c.y)
FROM VALUES NAMED_STRUCT('x', 'A', 'y', 1), NAMED_STRUCT('x', 'A', 'y', 2) AS t(c)
GROUP BY c.x
ORDER BY c.x
{code}

Error:
{code}
org.apache.spark.sql.AnalysisException: cannot resolve 'c.x' given input columns: [sum(c.y), x]; line 5 pos 9;
'Sort ['c.x ASC NULLS FIRST], true
+- Aggregate [c#0.x], [c#0.x AS x#2, sum(c#0.y) AS sum(c.y)#5L]
   +- SubqueryAlias t
      +- LocalRelation [c#0]
{code}
Bug: Allow Project to host outer references in scalar subqueries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/21 18:42
Affects Version/s: 3.2.0

Support Project to host outer references in subqueries, for example:
{code:sql}
SELECT (SELECT c1) FROM t
{code}

Currently, it will throw AnalysisException:
{code}
org.apache.spark.sql.AnalysisException: Expressions referencing the outer query are not supported outside of WHERE/HAVING clauses
{code}
Bug: NoClassDefFoundError for org.slf4j.impl.StaticLoggerBinder in org.apache.spark.Logging#isLog4j12 when using SLF4J/Logback 2.x
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/21 16:51
Affects Version/s: 3.1.2

When using SLF4J 2.x, I hit the following exception:

 
java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder
Caused by: java.lang.ClassNotFoundException: org.slf4j.impl.StaticLoggerBinder
 

This is because org.slf4j.impl.StaticLoggerBinder no longer exists in SLF4J 2.x (see [http://www.slf4j.org/codes.html#StaticLoggerBinder).] Ideally, Spark should not have a hard dependency on an SFL4J 1.x impl classes. 

 

Perhaps reflection or NoClassDefFoundError try-catch blocks could be used in the logger detection code, so both SLF4J 1.x and 2.x could be supported at runtime.
Bug: Refactor sixth set of 20 query execution errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:22
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the sixth set of 20.
{code:java}
noRecordsFromEmptyDataReaderError
fileNotFoundError
unsupportedSchemaColumnConvertError
cannotReadParquetFilesError
cannotCreateColumnarReaderError
invalidNamespaceNameError
unsupportedPartitionTransformError
missingDatabaseLocationError
cannotRemoveReservedPropertyError
namespaceNotEmptyError
writingJobFailedError
writingJobAbortedError
commitDeniedError
unsupportedTableWritesError
cannotCreateJDBCTableWithPartitionsError
unsupportedUserSpecifiedSchemaError
writeUnsupportedForBinaryFileDataSourceError
fileLengthExceedsMaxLengthError
unsupportedFieldNameError
cannotSpecifyBothJdbcTableNameAndQueryError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: support for creating table with void column datatype
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/21 09:34
Affects Version/s: 3.3.0

previously we blocked creating table with void column datatype to follow the hive behavior in PR: 

[https://github.com/apache/spark/pull/28833]

 

But according to the discussion here: [https://github.com/apache/spark/pull/28833#discussion_r613003850]

creating a table/view with void datatype is actually useful, so we need to restore the previous behvior
Bug: Refactor first set of 20 query compilation errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/21 09:11
Affects Version/s: 3.2.0

Refactor some exceptions in QueryCompilationErrors to use error classes.

There are currently ~408 exceptions in this file; so this PR only focuses on the second set of 20.

{code:java}
nestedGeneratorError
moreThanOneGeneratorError
generatorOutsideSelectError
legacyStoreAssignmentPolicyError
unresolvedUsingColForJoinError
dataTypeMismatchForDeserializerError
fieldNumberMismatchForDeserializerError
upCastFailureError
unsupportedAbstractDataTypeForUpCastError
outerScopeFailureForNewInstanceError
referenceColNotFoundForAlterTableChangesError
windowSpecificationNotDefinedError
selectExprNotInGroupByError
groupingMustWithGroupingSetsOrCubeOrRollupError
pandasUDFAggregateNotSupportedInPivotError
aggregateExpressionRequiredForPivotError
writeIntoTempViewNotAllowedError
expectTableOrPermanentViewNotTempViewError
readNonStreamingTempViewError
viewDepthExceedsMaxResolutionDepthError
{code}
Bug: Refactor fourth set of 20 query parsing errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:45
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryParsingErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala] to use error classes.

There are currently ~100 exceptions in this file; so this PR only focuses on the fourth set of 20.

{code}
showFunctionsUnsupportedError
duplicateCteDefinitionNamesError
sqlStatementUnsupportedError
unquotedIdentifierError
duplicateClausesError
duplicateKeysError
unexpectedFomatForSetConfigurationError
invalidPropertyKeyForSetQuotedConfigurationError
invalidPropertyValueForSetQuotedConfigurationError
unexpectedFormatForResetConfigurationError
intervalValueOutOfRangeError
invalidTimeZoneDisplacementValueError
createTempTableNotSpecifyProviderError
rowFormatNotUsedWithStoredAsError
useDefinedRecordReaderOrWriterClassesError
directoryPathAndOptionsPathBothSpecifiedError
unsupportedLocalFileSchemeError
invalidGroupingSetError
{code}

For more detail, see the parent ticket SPARK-36094.
Bug: Refactor third set of 20 query parsing errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:44
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryParsingErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala] to use error classes.

There are currently ~100 exceptions in this file; so this PR only focuses on the third set of 20.

{code}
fromToIntervalUnsupportedError
mixedIntervalUnitsError
dataTypeUnsupportedError
partitionTransformNotExpectedError
tooManyArgumentsForTransformError
notEnoughArgumentsForTransformError
invalidBucketsNumberError
invalidTransformArgumentError
cannotCleanReservedNamespacePropertyError
propertiesAndDbPropertiesBothSpecifiedError
fromOrInNotAllowedInShowDatabasesError
cannotCleanReservedTablePropertyError
duplicatedTablePathsFoundError
storedAsAndStoredByBothSpecifiedError
operationInHiveStyleCommandUnsupportedError
operationNotAllowedError
descColumnForPartitionUnsupportedError
incompletePartitionSpecificationError
computeStatisticsNotExpectedError
addCatalogInCacheTableAsSelectNotAllowedError
{code}

For more detail, see the parent ticket SPARK-36094.
Bug: Refactor second set of 20 query parsing errors to use error classes
Status: Open | Priority: Major | Resolution: 
Created: 26/Jul/21 19:44
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryParsingErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala] to use error classes.

There are currently ~100 exceptions in this file; so this PR only focuses on the first set of 20.

{code}
repetitiveWindowDefinitionError
invalidWindowReferenceError
cannotResolveWindowReferenceError
joinCriteriaUnimplementedError
naturalCrossJoinUnsupportedError
emptyInputForTableSampleError
tableSampleByBytesUnsupportedError
invalidByteLengthLiteralError
invalidEscapeStringError
trimOptionUnsupportedError
functionNameUnsupportedError
cannotParseValueTypeError
cannotParseIntervalValueError
literalValueTypeUnsupportedError
parsingValueTypeError
invalidNumericLiteralRangeError
moreThanOneFromToUnitInIntervalLiteralError
invalidIntervalLiteralError
invalidIntervalFormError
invalidFromToUnitValueError
{code}

For more detail, see the parent ticket SPARK-36094.
Bug: Refactor first set of 20 query parsing errors to use error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jul/21 23:24
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryParsingErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala] to use error classes.

There are currently ~100 exceptions in this file; so this PR only focuses on the first set of 20.

{code}
invalidInsertIntoError
insertOverwriteDirectoryUnsupportedError
columnAliasInOperationNotAllowedError
emptySourceForMergeError
unrecognizedMatchedActionError
insertedValueNumberNotMatchFieldNumberError
unrecognizedNotMatchedActionError
mergeStatementWithoutWhenClauseError
nonLastMatchedClauseOmitConditionError
nonLastNotMatchedClauseOmitConditionError
emptyPartitionKeyError
combinationQueryResultClausesUnsupportedError
distributeByUnsupportedError
transformNotSupportQuantifierError
transformWithSerdeUnsupportedError
lateralWithPivotInFromClauseNotAllowedError
lateralJoinWithNaturalJoinUnsupportedError
lateralJoinWithUsingJoinUnsupportedError
unsupportedLateralJoinTypeError
invalidLateralJoinRelationError
{code}

For more detail, see the parent ticket SPARK-36094.
Bug: DESC TABLE in v2 show more detail information
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/21 02:13
Affects Version/s: 3.2.0

according to https://issues.apache.org/jira/browse/SPARK-36086?focusedCommentId=17383195&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17383195
Bug: Re-enable the test of external listerners in SparkSQLEnvSuite
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 25/Jul/21 06:28
Affects Version/s: 3.0.3

While we're trying to recover GA in branch 3.0 ([https://github.com/apache/spark/pull/33502),] {{org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite}} continues to fail in GA:

 

[info] - SPARK-29604 external listeners should be initialized with Spark classloader *** FAILED *** (34 seconds, 465 milliseconds)
[info] scala.Predef.refArrayOps[org.apache.spark.sql.util.QueryExecutionListener](session.get.listenerManager.listListeners()).exists(((x$1: org.apache.spark.sql.util.QueryExecutionListener) => x$1.isInstanceOf[test.custom.listener.DummyQueryExecutionListener])) was false (SparkSQLEnvSuite.scala:57)

 

It looks not a memory issue. I also ran it locally and it passed. Not sure why GA is special for the test.

 

Ignored the test for now to recover GA. We need to investigate why the test fails and recover it.
Bug: Spark ThriftServer does not search hive external table which storage in hbase
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 26/Jul/21 01:34
Affects Version/s: 3.0.1

hive version is 2.3.3, hbase version is 2.4.2;

CREATE EXTERNAL TABLE `temp.mapping_from_hbase`(
 `tuid` string COMMENT '',
 `index` int COMMENT '')
 ROW FORMAT SERDE
 'org.apache.hadoop.hive.hbase.HBaseSerDe'
 STORED BY
 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
 WITH SERDEPROPERTIES (
 'hbase.columns.mapping'=':key,info:index',
 'serialization.format'='1')
 TBLPROPERTIES (
 'hbase.mapred.output.outputtable'='x:tuid_to_index',
 'hbase.table.name'='x:tuid_to_index',
 'transient_lastDdlTime'='1626835726')

When I use spark version 2.4.3 to search temp.mapping_from_hbase which can find it, but use spark version 3.0.1 can not do it.
 !screenshot-1.png|width=1179,height=457!
Bug: FileNotFoundException from the shuffle push can cause the executor to terminate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/21 05:31
Affects Version/s: 3.1.0

Once the shuffle is cleaned up by the {{ContextCleaner}}, the shuffle files are deleted by the executors. In this case, the push of the shuffle data by the executors can throw {{FileNotFoundException}}. When this exception is thrown from the {{shuffle-block-push-thread}}, it causes the executor to fail. This is because of the default uncaught exception handler for Spark daemon threads which terminates the executor when there are uncaught exceptions for the daemon threads.
{code:java}
21/06/17 16:03:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[block-push-thread-1,5,main]
java.lang.Error: java.io.IOException: Error in opening FileSegmentManagedBuffer

{file=********/application_1619720975011_11057757/blockmgr-560cb4cf-9918-4ea7-a007-a16c5e3a35fe/0a/shuffle_1_690_0.data, offset=10640, length=190}
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1155)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Error in opening FileSegmentManagedBuffer\{file=*******/application_1619720975011_11057757/blockmgr-560cb4cf-9918-4ea7-a007-a16c5e3a35fe/0a/shuffle_1_690_0.data, offset=10640, length=190}

at org.apache.spark.network.buffer.FileSegmentManagedBuffer.nioByteBuffer(FileSegmentManagedBuffer.java:89)
at org.apache.spark.shuffle.ShuffleWriter.sliceReqBufferIntoBlockBuffers(ShuffleWriter.scala:294)
at org.apache.spark.shuffle.ShuffleWriter.org$apache$spark$shuffle$ShuffleWriter$$sendRequest(ShuffleWriter.scala:270)
at org.apache.spark.shuffle.ShuffleWriter.org$apache$spark$shuffle$ShuffleWriter$$pushUpToMax(ShuffleWriter.scala:191)
at org.apache.spark.shuffle.ShuffleWriter$$anon$2$$anon$4.run(ShuffleWriter.scala:244)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
... 2 more
Caused by: java.io.FileNotFoundException: ******/application_1619720975011_11057757/blockmgr-560cb4cf-9918-4ea7-a007-a16c5e3a35fe/0a/shuffle_1_690_0.data (No such file or directory)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.apache.spark.network.buffer.FileSegmentManagedBuffer.nioByteBuffer(FileSegmentManagedBuffer.java:62)
{code}
We can address the issue by handling "FileNotFound" exceptions in the push threads and netty threads by stopping the push when {{FileNotFound}} is encountered.
Bug: Fix equality comparison of unordered Categoricals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/21 17:28
Affects Version/s: 3.2.0

We cannot rely on codes when compare equality of unordered Categoricals.

An example looks like
{code:java}
>>> (ps.Series(pd.Categorical(list('abca'))) == ps.Series(pd.Categorical(list('bcaa'), categories=list('bca')))).sort_index()
0     True
1     True
2     True
3    False
dtype: bool
{code}
Bug: Upgrade ZSTD-JNI to 1.5.0-4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/21 18:35
Affects Version/s: 3.2.0

ZSTD-JNI 1.5.0-3 has a packaging issue.

https://github.com/luben/zstd-jni/issues/181#issuecomment-885138495
Bug: Updated the version of TimestampNTZ related changes as 3.3.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/21 06:29
Affects Version/s: 3.3.0

As we decided to release TimestampNTZ type in Spark 3.3, we should update the versions of TimestampNTZ related changes as 3.3.0.
Bug: Upgrade lz4-java to 1.8.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/21 06:14
Affects Version/s: 3.3.0

lz4-java 1.8.0 was released, which includes not only performance improvement but also Darwin aarch64 support.
https://github.com/lz4/lz4-java/releases/tag/1.8.0
https://github.com/lz4/lz4-java/blob/1.8.0/CHANGES.md
Bug: Support new functions make_timestamp_ntz and make_timestamp_ltz
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 08:30
Affects Version/s: 3.3.0

Syntax:
make_timestamp_ntz(year, month, day, hour, min, sec)  Create local date-time from year, month, day, hour, min, sec fields
make_timestamp_ltz(year, month, day, hour, min, sec, [timezone])  Create current timestamp with local time zone from year, month, day, hour, min, sec and timezone fields
Bug: Spark doesn’t support reading/writing TIMESTAMP_NTZ with ORC
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 08/Jul/21 09:46
Affects Version/s: 3.3.0

The OrcTimestamp of MapReduce library is instant of java.sql.Timestamp. As it is the only support Timestamp type of ORC, Spark doesn’t support reading/writing TIMESTAMP_NTZ with ORC.
Bug: Rename the type name of TimestampNTZType as "timestamp_ntz"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 09:00
Affects Version/s: 3.3.0

Rename the type name string of TimestampNTZType from "timestamp without time zone" to "timestamp_ntz".
This is to make the column header shorter and simpler.
Snowflake and Flink uses similar approach:
https://docs.snowflake.com/en/sql-reference/data-types-datetime.html
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/concepts/timezone/
Bug: Rename TimestampWithoutTZType to TimestampNTZType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 06:42
Affects Version/s: 3.3.0

The time name of `TimestampWithoutTZType` is verbose.  Rename it as `TimestampNTZType` so that
1.  it is easier to read and type.
2. As we have the function to_timestamp_ntz, this makes the names consistent.
3. We will introduce a new SQL configuration `spark.sql.timestampType` for the default timestamp type. The configuration values can be "TIMESTMAP_NTZ" or "TIMESTMAP_LTZ" for simplicity.
Bug: Support ANSI SQL LOCALTIMESTAMP datetime value function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 03:47
Affects Version/s: 3.3.0

Returns the current timestamp at the start of query evaluation as TIMESTAMP WITH OUT TIME ZONE. This is similar to "current_timestamp()"
Note we need to update the optimization rule ComputeCurrentTime so that  Spark returns the same result in a single query if the function is called multiple times
Bug: Return different timestamp literals based on the default timestamp type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 17:49
Affects Version/s: 3.3.0

For the timestamp literal, it should have following behavior.
1. When spark.sql.timestampType is TIMESTAMP_NTZ: if there is no time zone part, return timestamp without time zone literal; otherwise, return timestamp with local time zone literal

2. When spark.sql.timestampType is TIMESTAMP_LTZ: return timestamp with local time zone literal