av9ash/GitBugs huggingface dataset:
Bug: Support non-reserved  keyword TIMESTAMP_NTZ
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 17:46
Affects Version/s: 3.3.0

Support new keyword TIMESTAMP_NTZ, which can be used for:
* timestamp without time zone data type in DDL
* timestamp without time zone data type in Cast clause.
* timestamp without time zone data type literal
Bug: Assign pretty SQL string to TimestampNTZ literals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 16:51
Affects Version/s: 3.3.0

Currently the TimestampNTZ literals shows only long value instead of timestamp string in its SQL string and toString result.
Before changes (with default timestamp type as TIMESTAMP_NTZ)
```
 -- !query
 select timestamp '2019-01-01\t'
 -- !query schema
struct<1546300800000000:timestamp_ntz>
```

After changes:
 -- !query
 select timestamp '2019-01-01\t'
 -- !query schema
struct<TIMESTAMP_NTZ '2019-01-01 00:00:00':timestamp_ntz>
Bug: Add new SQL function to_timestamp_ltz
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/21 07:37
Affects Version/s: 3.3.0

As the result of to_timestamp become consistent with the SQL configuration spark.sql.timestmapType and there is already a SQL function to_timestmap_ntz, we need  new function to_timestamp_ltz to construct timestamp with local time zone literal.
syntax:
```
to_timestamp_ltz(timestamp_str_column[, fmt])
to_timestamp_ltz(timestamp_column)
to_timestamp_ltz(date_column)
```
Bug: Support non-reserved  keyword TIMESTAMP_LTZ
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 17:47
Affects Version/s: 3.3.0

Support new keyword TIMESTAMP_LTZ, which can be used for:
* timestamp with local time zone data type in DDL
* timestamp with local time zone data type in Cast clause.
* timestamp with local time zone data type literal
Bug: New configuration spark.sql.timestampType for the default timestamp type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 16:10
Affects Version/s: 3.3.0

Add a new configuration `spark.sql.timestampType`, which configures the default timestamp type of Spark SQL, including SQL DDL and Cast clause. Setting the configuration as TIMESTAMP_NTZ will use TIMESTAMP WITHOUT TIME ZONE as the default type while putting it as TIMESTAMP_LTZ will use TIMESTAMP WITH LOCAL TIME ZONE.
Bug: Optimize OneRowRelation subqueries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 22:37
Affects Version/s: 3.2.0

Inline subqueries with OneRowRelation as leaf nodes instead of decorrelating and rewriting them as left outer joins.

Scalar subquery:
 ```
 SELECT (SELECT c1) FROM t1 -> SELECT c1 FROM t1
 ```

Lateral subquery:
 ```
 SELECT * FROM t1, LATERAL (SELECT c1, c2) -> SELECT c1, c2 , c1, c2 FROM t1
 ```
Bug: Step by days in the Sequence expression for dates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 06:54
Affects Version/s: 3.2.0

Allow to generate a sequence of dates by day step in a range of dates. For instance:
{code:sql}
spark-sql> select sequence(date'2021-07-01', date'2021-07-10', interval '3' day);
Error in query: cannot resolve 'sequence(DATE '2021-07-01', DATE '2021-07-10', INTERVAL '3' DAY)' due to data type mismatch:
sequence uses the wrong parameter type. The parameter type must conform to:
1. The start and stop expressions must resolve to the same type.
2. If start and stop expressions resolve to the 'date' or 'timestamp' type
then the step expression must resolve to the 'interval' or
'interval year to month' or 'interval day to second' type,
otherwise to the same type as the start and stop expressions.
         ; line 1 pos 7;
'Project [unresolvedalias(sequence(2021-07-01, 2021-07-10, Some(INTERVAL '3' DAY), Some(Europe/Moscow)), None)]
+- OneRowRelation
{code}
Bug: Null-based filter estimates should always be non-negative
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/21 22:57
Affects Version/s: 3.2.0

It's possible for a column's statistics to have a higher `nullCount` than the table's `rowCount`. In this case, the filter estimates come back outside of the reasonable range (between 0 and 1).
Bug: Interval str should handle start field == end Field
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/21 02:15
Affects Version/s: 3.2.0

{code:java}
-- !query
SELECT INTERVAL '-9223372036854.775808' SECOND
-- !query schema
struct<INTERVAL '-2562047788 hours -54.775808 seconds':interval>
-- !query output
-2562047788 hours -54.775808 seconds
{code}
Bug: Add flag to allow Driver to request for OPPORTUNISTIC containers
Status: Open | Priority: Major | Resolution: 
Created: 20/Jul/21 03:22
Affects Version/s: 3.3.0

YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes and specifically OPPORTUNISTIC containers.
The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to provide hints via config to the SPARK framework as to the number of containers it would like to schedule as OPPORTUNISTIC.


like MAPREDUCE-6703 .
Bug: jackson-databind RCE vulnerability
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 14/Jul/21 06:50
Affects Version/s: 3.1.2

Need to upgrade   jackson-databind version to *2.9.3.1*

At the beginning of 2018, jackson-databind was reported to contain another remote code execution (RCE) vulnerability (CVE-2017-17485) that affects versions 2.9.3 and earlier, 2.7.9.1 and earlier, and 2.8.10 and earlier. This vulnerability is caused by jackson-dababind’s incomplete blacklist. An application that uses jackson-databind will become vulnerable when the enableDefaultTyping method is called via the ObjectMapper object within the application. An attacker can thus compromise the application by sending maliciously crafted JSON input to gain direct control over a server. Currently, a proof of concept (POC) exploit for this vulnerability has been publicly available. All users who are affected by this vulnerability should upgrade to the latest versions as soon as possible to fix this issue.
Bug: Incorrect parsing of the start field in interval literals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/21 19:44
Affects Version/s: 3.2.0

The example below portraits the issue:

{code:sql}
spark-sql> select interval '123456:12' minute to second;
Error in query:
requirement failed: Interval string must match day-time format of '^(?<sign>[+|-])?(?<minute>\d{1,2}):(?<second>(\d{1,2})(\.(\d{1,9}))?)$': 123456:12, set spark.sql.legacy.fromDayTimeString.enabled to true to restore the behavior before Spark 3.0.(line 1, pos 16)

== SQL ==
select interval '123456:12' minute to second
----------------^^^
{code}
The start field can have more than 2 digits, actually.
Bug: Support for specifiying executor/driver node selector
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/21 16:51
Affects Version/s: 3.3.0

Now we can only use "spark.kubernetes.node.selector" to set lable for executor/driver. Sometimes, we need set executor/driver pods to different selector separately.

We can add below configure to enable the support for specifiying executor/driver node selector:

- spark.kubernetes.driver.node.selector.

- spark.kubernetes.executor.node.selector.
Bug: Do not remove REPARTITION_BY_NUM shuffle if AQE is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/21 09:53
Affects Version/s: 3.2.0

The shuffle origin is `REPARTITION_BY_NUM` if user specify an exact partition number with repartition, then we should not do any change of the number. That said, the shuffle output partitioning number should be always same with user expected.
Bug: Change quoted interval literal (interval constructor) to be converted to ANSI interval types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 01:53
Affects Version/s: 3.2.0

The tnit-to-unit interval literals and the unit list interval literals are converted to ANSI interval types but quoted interval literals are still converted to CalendarIntervalType.

{code}
-- Unit list interval literals
spark-sql> select interval 1 year 2 month;
1-2
-- Quoted interval literals
spark-sql> select interval '1 year 2 month';
1 years 2 months
{code}
Bug: Support explain final plan in AQE
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Jul/21 04:37
Affects Version/s: 3.3.0

The executed plan will be changed during running in AQE, however the current implementation of explain does not support this.

As the AQE is enabled by default, user may want to get the final plan using query, so it should make sense to add a new grammar to support it.
Bug: Upgrade GenJavadoc to 0.18
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 02:20
Affects Version/s: 3.2.0

GenJavadoc 0.18 was released, which includes a bug fix for Scala 2.13.
https://github.com/lightbend/genjavadoc/releases/tag/v0.18
Bug: Missing validation of regexp_replace inputs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/21 04:36
Affects Version/s: 3.3.0

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala overrides checkInputDataTypes, but it doesn't call super.checkInputDataTypes, so basic type checking is disabled.

 
{code:java}
scala> spark.sql("""select regexp_replace(collect_list(1), "1", "2")""").collect()
221/07/14 20:58:38 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 72, Column 1: Assignment conversion not possible from type "org.apache.spark.sql.catalyst.util.ArrayData" to type "org.apache.spark.unsafe.types.UTF8String"
3org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 72, Column 1: Assignment conversion not possible from type "org.apache.spark.sql.catalyst.util.ArrayData" to type "org.apache.spark.unsafe.types.UTF8String"
4	at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12021)
{code}
Bug: dayofweek documentation for python and R
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/21 06:27
Affects Version/s: 2.3.0

Python and R documentation for {{dayofweek}} doesn't mention which integer corresponds to which day. The information is only available in {{sql/core/src/main/scala/org/apache/spark/sql/functions.scala}}
Bug: [SQL] ArrayIndexOutOfBounds in CAST string to timestamp
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/21 18:04
Affects Version/s: 3.0.3

I discovered this bug during some fuzz testing.
{code:java}
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.1
      /_/
         
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.scala> 

scala> import org.apache.spark.sql.types.DataTypes

scala> val df = Seq(":8:434421+ 98:38").toDF("c0")
df: org.apache.spark.sql.DataFrame = [c0: string]

scala> val df2 = df.withColumn("c1", col("c0").cast(DataTypes.TimestampType))
df2: org.apache.spark.sql.DataFrame = [c0: string, c1: timestamp]

scala> df2.show
java.lang.ArrayIndexOutOfBoundsException: 9
  at org.apache.spark.sql.catalyst.util.DateTimeUtils$.stringToTimestamp(DateTimeUtils.scala:328)
  at org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToTimestamp$2(Cast.scala:455)
  at org.apache.spark.sql.catalyst.expressions.CastBase.buildCast(Cast.scala:295)
  at org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToTimestamp$1(Cast.scala:451)
  at org.apache.spark.sql.catalyst.expressions.CastBase.nullSafeEval(Cast.scala:840)
  at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:476)
 {code}
Bug: Support for Slowly Changing Dimension Build Or Data Updates
Status: Open | Priority: Major | Resolution: 
Created: 14/Jul/21 21:01
Affects Version/s: 3.0.3

After multiple searches raising this request.

Understand the storage is cheap these days , but will it be not useful to allow a build of Slowly Changing Dimension Type 2 with history , native to the Spark Software with update and insert on any datapoint.
Bug: spark function from_json should output field name, field type and field value when FAILFAST mode throw exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/21 08:18
Affects Version/s: 3.0.0

spark function from_json outputs error message when FAILFAST mode throw exception.

 

But the message does not contain important info exemaple: field name, field vlue , field type...

 

This  infoormation is very important for devlops to find where error input data is located.
Bug: Fix flaky KafkaSourceStressSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/21 01:28
Affects Version/s: 3.0.3

The test logic in KafkaSourceStressSuite is not stable. For example, [https://github.com/apache/spark/runs/3049244904.]

Once we add data to a topic and then delete the topic before checking data, the expected answer is different to retrieved data from the sink.
Bug: Update the document about the behavior change of trimming characters for cast
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jul/21 19:59
Affects Version/s: 3.0.3

sql-migration-guide.md mentions about the behavior of cast like as follows.
{code}
In Spark 3.0, when casting string value to integral types(tinyint, smallint, int and bigint), datetime types(date, timestamp and interval) and boolean type, the leading and trailing whitespaces (<= ASCII 32) will be trimmed before converted to these type values, for example, `cast(' 1\t' as int)` results `1`, `cast(' 1\t' as boolean)` results `true`, `cast('2019-10-10\t as date)` results the date value `2019-10-10`. In Spark version 2.4 and below, when casting string to integrals and booleans, it does not trim the whitespaces from both ends; the foregoing results is `null`, while to datetimes, only the trailing spaces (= ASCII 32) are removed.
{code}

In fact,  select cast('2019-10-10\b' as date); returns 2019-10-10 in Spark 3.0.0. 
But SPARK-32559 changed this behavior and since 3.0.1, the query returns NULL.
Bug: Join can become unresolved after PullupCorrelatedPredicates
Status: Open | Priority: Major | Resolution: 
Created: 13/Jul/21 04:45
Affects Version/s: 3.2.0

Join can become unresolved after PullupCorrelatedPredicates:
{code:sql}
create view t1(c1, c2) as values (0, 1), (1, 2)
create view t2(c1, c2) as values (0, 2), (0, 3)

select (
  select sum(l.cnt + r.cnt)
  from (select count(*) cnt from t2 where t1.c1 = t2.c1 having cnt = 0) l
  join (select count(*) cnt from t2 where t1.c1 = t2.c1 having cnt = 0) r
  on l.cnt = r.cnt
) from t1

== Optimized Logical Plan ==
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '(' expecting {<EOF>, '.', '-'}(line 1, pos 14)

== SQL ==
scalarsubquery(c1, c1)
--------------^^^
{code}

This is because duplicate attributes are not handled correctly when pulling up correlated predicates over joins. Both `pullOutCorrelatedPredicates` and `DecorrelateInnerQuery` are subject to this issue.
Bug: ResolveSubquery should skip unary node UnresolvedHaving
Status: Open | Priority: Major | Resolution: 
Created: 13/Jul/21 04:40
Affects Version/s: 3.2.0

An UnresolvedHaving operator must be resolved by ResolveAggregateFunction first before resolving subqueries in it. Please see [https://github.com/apache/spark/pull/32470/files#r650172832] for more details.
Bug: Consolidate tests for data-type-based operations of decimal Series
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/21 22:56
Affects Version/s: 3.2.0

Tests for data-type-based operations of decimal Series are in two places:
 * python/pyspark/pandas/tests/data_type_ops/test_decimal_ops.py
 * python/pyspark/pandas/tests/data_type_ops/test_num_ops.py

We'd better either merge test_decimal_ops into test_num_ops or keep all tests related to decimal Series to test_decimal_ops.
Bug: Make broadcast query stage executionContext isolation from AQE
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 12/Jul/21 04:04
Affects Version/s: 3.1.2

`AdaptiveSparkPlanExec` shares the execution context with `BroadcastQueryStage`, but it has a small number thread (hard code 16). Unfortunately, it's not enough in some complex queries and no thread is available to cancel the broadcast stage.

 

In normal code path (without AQE), broadcast has it's own execution context with a config `spark.sql.broadcastExchange.maxThreadThreshold`. It should make sense to make broadcast query stage executionContext isolation from AQE and respect the existed thread threshold config.
Bug: An Impl of skew key detection and data inflation optimization
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jul/21 09:20
Affects Version/s: 3.3.0

design doc: [https://docs.google.com/document/d/1jYGlipQdirhuRR3_x1PflAYdMow7StOHOLUjSw6Vk9s/edit?usp=sharing]

1, introduce {{ShuffleExecAccumulator}} in ShuffleExchangeExec to support arbitrary statistics;

2, impl a key sampling {{ShuffleExecAccumulator}} to detect skew keys and show debug info on SparkUI;

3, in {{OptimizeSkewedJoin}}, estimate the joined size of each partition based on the sampled keys, and split a partition if it is not split yet and its estimated joined size is too larger.

 

data inflation case:
{code:java}
spark.conf.set("spark.sql.adaptive.enabled", true)
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", true)
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "80")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "80")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
spark.conf.set("spark.sql.adaptive.coalescePartitions.minPartitionNum", "1")
spark.conf.set("spark.sql.shuffle.partitions", 10)

sc.setLogLevel("INFO")

val df1 = spark.range(0, 1000000, 1, 9).select(col("id").as("key1"), col("id").as("value1"), hash(col("id")).mod(100).as("hash1")).withColumn("key1", when(col("hash1") === lit(0), lit(0)).otherwise(col("key1")))

val df2 = spark.range(0, 500000, 1, 7).select(col("id").as("key2"), col("id").as("value2"), hash(col("id")).mod(100).as("hash2")).withColumn("key2", when(col("hash2") === lit(0), lit(0)).otherwise(col("key2")))

df1.join(df2, col("key1") === col("key2")).write.mode("overwrite").parquet("/tmp/result_0")

spark.conf.set("spark.sql.adaptive.skewJoin.inflation.enabled", true)
spark.conf.set("spark.sql.adaptive.skewJoin.inflation.factor", 10)
spark.conf.set("spark.sql.adaptive.shuffle.sampleSizePerPartition", 100)
spark.conf.set("spark.sql.adaptive.shuffle.detectSkewness", true)

df1.join(df2, col("key1") === col("key2")).write.mode("overwrite").parquet("/tmp/result_1"){code}
 

it also partially resolve https://issues.apache.org/jira/browse/SPARK-35596
Bug: Complete mappings between numpy literals and Spark data types
Status: Open | Priority: Major | Resolution: 
Created: 09/Jul/21 18:23
Affects Version/s: 3.2.0

In [https://github.com/apache/spark/pull/32955,] the {{lit}} function defined in {{pyspark.pandas.spark.functions}} has been adjusted to support numpy literals input.

However, the mapping between numpy literals and Spark data types is not complete.

We ought to fill the gap.
Bug: Support numpy literals as input for pandas-on-Spark APIs 
Status: Open | Priority: Major | Resolution: 
Created: 09/Jul/21 18:20
Affects Version/s: 3.2.0

Some pandas-on-Spark APIs use PySpark column-related APIs internally, and these column-related APIs don't support numpy literals, thus numpy literals are disallowed as input (e.g. {{to_replace}} parameter in {{replace}} API). 

`isin` method has been adjusted in [https://github.com/apache/spark/pull/32955|https://github.com/apache/spark/pull/32955,] . We ought to adjust other API to support numpy literals.
Bug: Use inputPlan instead of currentPhysicalPlan to initialize logical link
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/21 09:17
Affects Version/s: 3.2.0

At {{initialPlan}} we may remove some Spark Plan with {{queryStagePreparationRules}}, if removed Spark Plan is top level node, then we will lose the linked logical node.

Since we support AQE side broadcast join config. It's more common that a join is SMJ at normal planner and changed to BHJ after AQE reOptimize. However, {{RemoveRedundantSorts}} is applied before reOptimize at {{initialPlan}}, then a local sort might be removed incorrectly if a join is SMJ at first but changed to BHJ during reOptimize.
Bug: Lost the null flag info when show create table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/21 05:16
Affects Version/s: 3.2.0

When execute `SHOW CREATE TALBE XXX` command, the ddl info lost the null flag.
{code:java}
// def toDDL: String = s"${quoteIdentifier(name)} ${dataType.sql}$getDDLComment"
{code}
Bug: [Dynamic allocation] Executor grace period (ExecutorIdleTimeout) ignored due to nulll startTime for pods in pending state
Status: Open | Priority: Major | Resolution: 
Created: 08/Jul/21 07:42
Affects Version/s: 3.1.1

Pending executor are always timeouted due to null startTime and funtion returning true in case of exception in parsing startTime.

 

In class ExecutorPodsAllocator:

{{private def isExecutorIdleTimedOut(state: ExecutorPodState, currentTime: Long): Boolean = {}}
 {{try {}}
 {{ val startTime = Instant.parse(state.pod.getStatus.getStartTime).toEpochMilli()}}
 {{ currentTime - startTime > executorIdleTimeout}}
 {{catch {}}
 {{  case _: Exception =>}}
 {{  logDebug(s"Cannot get startTime of pod ${state.pod}")}}}}
 {{  true}}
 }}
Bug: Respect interval fields in extract
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/21 19:53
Affects Version/s: 3.2.0

Extract should process only existing fields of interval types. For example:

{code:sql}
spark-sql> SELECT EXTRACT(MONTH FROM INTERVAL '2021-11' YEAR TO MONTH);
11
spark-sql> SELECT EXTRACT(MONTH FROM INTERVAL '2021' YEAR);
0
{code}
The last command should fail as the month field doesn't present in INTERVAL YEAR.
Bug: Add a config to force using ShuffledHashJoin for test purpose
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/21 02:43
Affects Version/s: 3.2.0

In the join.sql, we want to cover all 3 join types. but the problem is currently the `spark.sql.join.preferSortMergeJoin = false` couldn't guarantee all the joins will use ShuffledHashJoin, so we need another config to force using hash join in the testing.
Bug: Disallow altering permanent views based on temporary views or UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/21 04:45
Affects Version/s: 3.1.2

PR [#15764|https://github.com/apache/spark/pull/15764] disabled creating permanent views based on temporary views or UDFs. But AlterViewCommand didn't block temporary objects.