av9ash/GitBugs huggingface dataset:
Bug: Check existence before deleting stagingDir in HadoopMapReduceCommitProtocol
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Feb/22 10:55
Affects Version/s: 3.2.1

In `commitJob` and `abortJob` of HadoopMapReduceCommitProtocol, the stagingDir was deleted, but for cases like the following example, the stagingDir was not created in the first place.

If the underLayer FileSystem is HDFS, there won't be any logs, but deleting a nonexistent file with FileSystem.delete would also acquire the WRITE LOCK of NameNode, if such operations are frequent, it would hurt the performance of NN.
For other FileSystem, like Alluxio, a warning log was thrown as follows.
{code:java}
AbstractFileSystem: delete failed: alluxio.exception.FileDoesNotExistException: Path "/test/spark/t10/.spark-staging-3cfe0d7c-3749-44de-9da7-f811a40aa4af" does not exist. {code}
This ticket is to add an existence check before the actual deleting operation, which could help eliminate the WARN log or not hurt HDFS' performance.
Bug: Explain formatted does not collect subqueries under query stage in AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/22 12:07
Affects Version/s: 3.1.2

ExplainUtils have not catched QueryStageExec during collecting subquries. So we can not get the subqueries formatted explain who is under the QueryStageExec.

Note that, it also affects the subquery of dpp.
Bug: Adjust input `format` of function `to_binary`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/22 03:20
Affects Version/s: 3.3.0

Currently, function to_binary doesn't deal with the non-string `format` parameter properly.
For example, `spark.sql("select to_binary('abc', 1)")` raises casting error, rather than hint that encoding format is unsupported. 

In addition, `base2` format is arguable as discussed [here](https://github.com/apache/spark/pull/35415#discussion_r805578036). We may exclude it following what Snowflake [to_binary](https://docs.snowflake.com/en/sql-reference/functions/to_binary.html) does for now.
Bug: Func(wrong data type) in HAVING clause should throw data mismatch error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/22 07:09
Affects Version/s: 3.3.0

{code:java}
with t as (select true c)
3select t.c
4from t
5group by t.c
6having mean(t.c) > 0 {code}
This query throws "Column 't.c' does not exist. Did you mean one of the following? [t.c]"

However, mean(boolean) is not a supported function signature, thus error result should be  "cannot resolve 'mean(t.c)' due to data type mismatch: function average requires numeric or interval types, not boolean"

 

This is because
 # The mean(boolean) in HAVING was not marked as resolved in {{ResolveFunctions}} rule.

 # Thus in {{{}ResolveAggregationFunctions{}}}, the {{TempResolvedColumn}} as a wrapper in mean({{{}TempResolvedColumn{}}}(t.c)) cannot be removed (only resolved AGG can remove its’s TempResolvedColumn).

 # Thus in a later batch rule applying,  {{TempResolvedColumn}} was reverted and it becomes mean(`t.c`), so mean loses the information about t.c.

 # Thus at the last step, the analyzer can only report t.c not found.

 

mean(boolean) in HAVING is not marked as resolved in {{ResolveFunctions}} rule because 
 # It uses Expression default `resolved` field population code 
{code:java}
lazy val resolved: Boolean = childrenResolved && checkInputDataTypes().isSuccess {code}
 
 #  During the analyzing,  mean(boolean) is mean(TempResolveColumn(boolean), thus childrenResolved is true.
 # however checkInputDataTypes() will be false ([Average.scala#L55|[https://github.com/apache/spark/blob/74ebef243c18e7a8f32bf90ea75ab6afed9e3132/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala#L55])]
 # Thus eventually Average's `resolved`  will be false, but it leads to wrong error message.
Bug: Fix NoSuchElementException if pushed filter does not contain any references
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/22 04:51
Affects Version/s: 3.3.0

reproduce:

{code:java}
CREATE TABLE t (c1 int) USING PARQUET;

SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.BooleanSimplification;

SELECT * FROM t WHERE c1 = 1 AND 2 > 1;
{code}

and the error msg:

{code:java}
java.util.NoSuchElementException: next on empty iterator
	at scala.collection.Iterator$$anon$2.next(Iterator.scala:41)
	at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
	at scala.collection.mutable.LinkedHashSet$$anon$1.next(LinkedHashSet.scala:89)
	at scala.collection.IterableLike.head(IterableLike.scala:109)
	at scala.collection.IterableLike.head$(IterableLike.scala:108)
	at org.apache.spark.sql.catalyst.expressions.AttributeSet.head(AttributeSet.scala:69)
	at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.$anonfun$listFiles$3(PartitioningAwareFileIndex.scala:85)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.listFiles(PartitioningAwareFileIndex.scala:84)
	at org.apache.spark.sql.execution.FileSourceScanExec.selectedPartitions$lzycompute(DataSourceScanExec.scala:249)
{code}
Bug: Contains Join for Spark SQL
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Feb/22 09:06
Affects Version/s: 3.3.0

Currently Spark SQL uses a Broadcast Nested Loop join when it has to execute the following string contains query:
{code:sql}
SELECT a.text, b.pattern
FROM fact_table a
JOIN patterns b
ON a.text like concat('%', b.pattern, '%');
{code}
OR
{code:sql}
SELECT a.text, b.pattern
FROM fact_table a
JOIN patterns b
ON position(b.pattern, a.text) > 0;
{code}
If there are many patterns to match in the left table, the query many execute for a long time.

Actually this kind of join is called *Multi-Pattern String Matching* or {*}Multi-Way String Matching{*}, and many algorithms try to improve this kind of matching. One of the well-knowing algorithm is [*Aho–Corasick algorithm*|https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm]

The basic idea of optimizing this kindof query is to transform all the patterns into a trie tree and broadcast it. So each row of the fact table only need to match its content against the trie tree once.

The query will go from *O(M * N * m * n)* to *O(M * m * max( n ))*
M = number of records in the fact table
N = number of records in the patterns table
m = row length of the fact table
n = row length of the patterns table
Bug: Provide monitoring REST API for Structured Streaming
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Feb/22 20:40
Affects Version/s: 3.3.0

In [SPARK-38234|https://issues.apache.org/jira/browse/SPARK-38234] {-}{-}Structured Streaming is added to the history server and a "Structured Streaming" tab appears in the history UI when a streaming query is present. However, even though a store exists for it and the data is presented in the UI, this data is not exposed as a REST API. This data can be used for monitoring, detecting streaming and to build custom dashboards. This monitoring API will be similar to the monitoring APIs that are present for DStreams - refer [SPARK-18470|https://issues.apache.org/jira/browse/SPARK-18470].

In this change, we plan to add two simple APIs that expose the data in the store and can be used to monitor streaming queries. 
h3. *Summary API*

To list the summary of all existing streaming queries.

GET {{/\{appId}/sql/streamingqueries}}

Response is list of {_}StreamingQueryData{_}.
h3. *Progress API*

To list the progress events of a specific streaming query by {_}runId{_}. 

User can also specify how many of the most recent events needs to be retrieved by using the _last_ query parameter. By default, we can return the most recent progress event i.e. last is set to 1.

GET {{{}/\{appId}/sql/streamingqueries/\{runId}/progress?last={N{}}}}

Response is list of {_}StreamingQueryProgress{_}.

*Note:* We are not introducing new objects for the response since we are just returning the data from the store without aggregation, these are existing event structures.

Attached sample I/O and screenshots.{{{{}}{}}}
Bug: Expose Node Description attribute in SQL Rest API
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Feb/22 23:04
Affects Version/s: 3.2.0

Currently, SQL public Rest API does not expose node description and it is useful to have nodeDesc attribute at query level to have more details such as:
{code:java}
- Join Operators(BHJ, SMJ, SHJ) => when correlating join operator with join type and which leg is built for BHJ. 
- HashAggregate => aggregated keys and agg functions
- List can be extended for other physical operators.{code}
*Current Sample Json Result:*
{code:java}
{
    "nodeId" : 14,
    "nodeName" : "BroadcastHashJoin",
    "wholeStageCodegenId" : 3,
    "stageIds" : [ 5 ],
    "metrics" : [ {
          "name" : "number of output rows",
          "value" : {
        "amount" : "2"
          }
    }
},
...
{
    "nodeId" : 8,
    "nodeName" : "HashAggregate",
    "wholeStageCodegenId" : 4,
    "stageIds" : [ 8 ],
    "metrics" : [ {
      "name" : "spill size",
      "value" : {
        "amount" : "0.0"
      }
    }
} {code}
*New* {*}Sample Json Result{*}{*}:{*}
{code:java}
{
    "nodeId" : 14,
    "nodeName" : "BroadcastHashJoin",
    "nodeDesc" : "BroadcastHashJoin [id#4], [id#24], Inner, BuildLeft, false",
    "wholeStageCodegenId" : 3,
    "stageIds" : [ 5 ],
    "metrics" : [ {
          "name" : "number of output rows",
          "value" : {
        "amount" : "2"
          }
    }
},
...
{
    "nodeId" : 8,
    "nodeName" : "HashAggregate",
    "nodeDesc" : "HashAggregate(keys=[name#5, age#6, salary#18], functions=[avg(cast(age#6 as bigint)), avg(salary#18)])",
    "wholeStageCodegenId" : 4,
    "stageIds" : [ 8 ],
    "metrics" : [ {
      "name" : "spill size",
      "value" : {
        "amount" : "0.0"
      }
    }
} {code}
Bug: The columns in state schema should be relaxed to be nullable
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/22 11:20
Affects Version/s: 3.1.2

Starting from SPARK-27237, Spark validates the schema of state across query runs to make sure it doesn't fall into more weird issue like SIGSEGV on the runtime.

The comparison logic is reasonable in terms of nullability; it has below matrices:
||existing schema||new schema||allowed||
|nullable|nullable|O|
|nullable|non-nullable|O|
|non-nullable|nullable|X|
|non-nullable|non-nullable|O|

What we miss here is, the nullability of the column can be changed in the optimizer (mostly nullable to non-nullable), and the optimization about nullability could be applied differently with any simple changes.

So this scenario is hypothetically possible:

1. At the first run of the query, optimizer marks some columns from nullable to non-nullable, and it goes to the schema of the state. (state schema has a column with non-nullable)
2. At the second run of the query (possibly with code modification or upgrading Spark version), optimizer no longer marks such columns from nullable to non-nullable, and it goes with comparison of the schema of the state (existing vs new), comparing non-nullable (existing) vs nullable (new), which is NOT allowed.

In terms of storage view for state store, it is not required to determine the column as non-nullable vs nullable. Interface-wise, state store has no concept of schema; so it is safe to relax such constraint, and open the chance for optimizer to do whatever it wants and doesn't break stateful operators.
Bug: array_sort does not allow non-orderable datatypes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/22 12:29
Affects Version/s: 3.2.1

 {{array_sort}} has check to see if the entries it has to sort are orderable.

I think this check should be removed.  Because even entries which are not orderable can have a lambda function which makes them orderable.
{code:java}
Seq((Array[Map[String, Int]](Map("a" -> 1), Map()), "x")).toDF("a", "b").selectExpr("array_sort(a, (x,y) -> cardinality(x) - cardinality(y))"){code}
fails with:
{code:java}
org.apache.spark.sql.AnalysisException: cannot resolve 'array_sort(`a`, lambdafunction((cardinality(namedlambdavariable()) - cardinality(namedlambdavariable())), namedlambdavariable(), namedlambdavariable()))' due to data type mismatch: array_sort does not support sorting array of type map<string,int> which is not orderable {code}
While the case where this check is relevant, fails with a different error which is triggered earlier in the code path:
{code:java}
> Seq((Array[Map[String, Int]](Map("a" -> 1), Map()), "x")).toDF("a", "b").selectExpr("array_sort(a)"){code}
Fails with:
{code:java}
org.apache.spark.sql.AnalysisException: cannot resolve '(namedlambdavariable() < namedlambdavariable())' due to data type mismatch: LessThan does not support ordering on type map<string,int>; line 1 pos 0;
{code}
Bug: Streaming dropDuplicates should remove out-of-watermark keys
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Feb/22 06:14
Affects Version/s: 3.3.0

Currently streaming dropDuplicates simply stores all states forever if no watermark-attached key exists.

For streaming stateful operator, this seems counterintuitive behavior if we consider other stateful operations such as streaming joining.

It also means the number of state will grow up infinitely as we observe in real application. But it doesn't make sense to add a time column into dedup keys because we don't really deduplicate rows by [key, time] but only by [key].

More reasonable streaming dedup seems to remove out-of-watermark states if the input data (not key) has watermark. This will be a behavior change, for streaming queries with dedup operation.
Bug: Handle `Pacific/Kanton` in DateTimeUtilsSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/22 03:14
Affects Version/s: 3.1.3

This issue aims to fix the flaky UT failures due to https://bugs.openjdk.java.net/browse/JDK-8274407 (Update Timezone Data to 2021c) and its backport commits that renamed 'Pacific/Enderbury' to 'Pacific/Kanton' in the latest Java 17.0.2, 11.0.14, and 8u311.

Rename Pacific/Enderbury to Pacific/Kanton.

**MASTER**
- https://github.com/dongjoon-hyun/spark/runs/5119322349?check_suite_focus=true
{code}
[info] - daysToMicros and microsToDays *** FAILED *** (620 milliseconds)
[info]   9131 did not equal 9130 Round trip of 9130 did not work in tz Pacific/Kanton (DateTimeUtilsSuite.scala:783)
{code}

**BRANCH-3.2**
- https://github.com/apache/spark/runs/5122380604?check_suite_focus=true
{code}
[info] - daysToMicros and microsToDays *** FAILED *** (643 milliseconds)
[info]   9131 did not equal 9130 Round trip of 9130 did not work in tz Pacific/Kanton (DateTimeUtilsSuite.scala:771)
{code}
Bug: 'Column' object is not callable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/22 15:23
Affects Version/s: 3.1.2

Hi guys, I have such simple dataframe and am trying to create one new column.

That its schema:

 

>>>> df_operation_event_sellers.schema 

Out[69]: StructType(List(StructField(id,StringType,true),StructField(account_id,StringType,true),StructField(p_tenant_id,StringType,true),StructField(vendor_id,StringType,true),StructField(amount,DecimalType(38,18),true),StructField(operation_type,StringType,true),StructField(reference_id,StringType,true),StructField(date,TimestampType,true),StructField(carrier_id,StringType,true),StructField(account_number,StringType,true),StructField(data_source,StringType,true),StructField(entity,StringType,true),StructField(ingestion_date,DateType,true),StructField(event_type,StringType,false),StructField(amount_new,DecimalType(38,18),true),StructField(date_new,IntegerType,true),StructField(row_num,IntegerType,true)))

 

>>> command to create the new column

df_operation_event_sellers= df_operation_event_sellers.withColumn('flag_first_selling',when(col('row_num') == 1,'YES').instead('NO'))



ISSUE >>>>>>>>>>>> TypeError: 'Column' object is not callable

 

What is happing?

ps. I created other columns the same way successfully
Bug: Selectively include EXTERNAL TABLE source files via REGEX
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/22 15:47
Affects Version/s: 3.3.0

https://issues.apache.org/jira/browse/HIVE-951

CREATE EXTERNAL TABLE should allow users to cherry-pick files via regular expression.
CREATE EXTERNAL TABLE was designed to allow users to access data that exists outside of Hive, and
currently makes the assumption that all of the files located under the supplied path should be included
in the new table. Users frequently encounter directories containing multiple
datasets, or directories that contain data in heterogeneous schemas, and it's often
impractical or impossible to adjust the layout of the directory to meet the requirements of
CREATE EXTERNAL TABLE. A good example of this problem is creating an external table based
on the contents of an S3 bucket.

One way to solve this problem is to extend the syntax of CREATE EXTERNAL TABLE
as follows:

CREATE EXTERNAL TABLE
...
LOCATION path [file_regex]
...
Bug: Improve the error for pivoting of unsupported value types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/22 10:31
Affects Version/s: 3.3.0

The error message from:

{code:scala}
  test("Improve the error for pivoting of unsupported value types") {
    trainingSales
      .groupBy($"sales.year")
      .pivot(struct(lower($"sales.course"), $"training"))
      .agg(sum($"sales.earnings"))
      .show(false)
  }
{code}

can confuse users:

{code:java}
The feature is not supported: literal for '[dotnet,Dummies]' of class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema.
org.apache.spark.SparkRuntimeException: The feature is not supported: literal for '[dotnet,Dummies]' of class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.literalTypeUnsupportedError(QueryExecutionErrors.scala:245)
	at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:99)
	at org.apache.spark.sql.RelationalGroupedDataset.$anonfun$pivot$2(RelationalGroupedDataset.scala:455)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
{code}
Need to improve the error message and make it more precise.

See https://github.com/apache/spark/pull/35302#discussion_r793629370
Bug: Fix data incorrect if aggregate function is empty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/22 06:58
Affects Version/s: 3.2.1

The group only condition should check if the aggregate expression is empty.

In DataFrame api, it is allowed to make a empty aggregations.

So the following query should return 1 rather than 0 because it's a global aggregate.
{code:java}
val emptyAgg = Map.empty[String, String]
spark.range(2).where("id > 2").agg(emptyAgg).limit(1).count
{code}
Bug: Fix wrong transformExpressions in Optimizer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Feb/22 14:30
Affects Version/s: 3.3.0

`transformExpressions` can only traverse all expressions in this current query plan, so the rule `EliminateDistinct` and `EliminateAggregateFilter` can not optimize the non-root node. We should use `transformAllExpressions` rather than `transformExpressions`.
Bug: ANSI mode: allow implicitly casting String to other simple types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Feb/22 12:31
Affects Version/s: 3.3.0

Compared to the default behavior, the current ANSI type coercion rules don't allow the following cases:
 * comparing String with other simple types, e.g. date/timestamp/int ...
 * arithmetic op with String and other simple types
 * Union/Intersect/Except with String and other simple types
 * SQL function expects non-string types but got  string input
 * other SQL operators..

The original purpose is to prevent potential String parsing errors under ANSI mode. However, after doing research among real-world Spark SQL queries, I find that many users are actually using String as Date/Timestamp/Numeric in their queries. 

 

To make the migration to ANSI mode easier, I suggest removing this limitation.
Bug: Correct the logic to measure the memory usage of RocksDB
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Feb/22 15:20
Affects Version/s: 3.2.1

Currently, structed streaming would report the memory usage of RocksDB as state store's metrics. However, it only sums from {{rocksdb.estimate-table-readers-mem}} and {{rocksdb.size-all-mem-tables}}. However, this lacks of the memory usage of block cache. (You can refer to [Memory-usage-in-RocksDB](https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB) for more details.)

BTW, as the "block-cache-pinned-usage" is included in "block-cache-usage", we don't need to include the pinned usage.
Bug: Use error classes in the execution errors of pivoting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 18:34
Affects Version/s: 3.3.0

Migrate the following errors in QueryExecutionErrors:
* repeatedPivotsUnsupportedError
* pivotNotAfterGroupByUnsupportedError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryExecutionErrorsSuite.
Bug: Fix /sql/hive-thriftserver/org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite under ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/22 05:55
Affects Version/s: 3.3.0

ThriftServerQueryTestSuite will fail on {{timestampNTZ/timestamp.sql}} , when ANSI mode is on by default. It is because the {{timestampNTZ/timestamp.sql}} should only work with ANSI off according to the golden result file, but ThriftServerQueryTestSuite or the timestamp.sql test doesn't override the default ANSI setting.
Bug: LikeSimplification handles escape character
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 09/Feb/22 14:36
Affects Version/s: 3.2.0

Currently, LikeSimplification rule of catalyst is skipped if the pattern contains escape character.

{noformat}
SELECT * FROM tbl LIKE '%100\%'

...
== Optimized Logical Plan ==
Filter (isnotnull(c_1#0) && c_1#0 LIKE %100\%)
+- Relation[c_1#0,c_2#1,c_3#2] ...
{noformat}
The filter LIKE '%100\%' in this query is not optimized into 'EndsWith' of StringType.

LikeSimplification rule can consider a special character(wildcard(%, _) or escape character) as a plain character if the character follows an escape character.
By doing that, LikeSimplification rule can optimize the filter like below.

{noformat}
SELECT * FROM tbl LIKE '%100\%'

...
== Optimized Logical Plan ==
Filter (isnotnull(c_1#0) && EndsWith(c_1#0, 100%))
+- Relation[c_1#0,c_2#1,c_3#2] 
{noformat}
Bug: Grouping by timestamp_ntz will sometimes corrupt the results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/22 22:00
Affects Version/s: 3.3.0

Assume this data:
{noformat}
create or replace temp view v1 as
select * from values
  (1, timestamp_ntz'2012-01-01 00:00:00', 10000),
  (2, timestamp_ntz'2012-01-01 00:00:00', 20000),
  (1, timestamp_ntz'2012-01-01 00:00:00', 5000),
  (1, timestamp_ntz'2013-01-01 00:00:00', 48000),
  (2, timestamp_ntz'2013-01-01 00:00:00', 30000)
  as data(a, b, c);
{noformat}
Run the following query:
{noformat}
select *
from v1
pivot (
  sum(c)
  for a in (1, 2)
);
{noformat}
You get incorrect results for the group-by column:
{noformat}
2012-01-01 19:05:19.476736	15000	20000
2013-01-01 19:05:19.476736	48000	30000
Time taken: 2.65 seconds, Fetched 2 row(s)
{noformat}
Actually, _whenever_ the TungstenAggregationIterator is used to group by a timestamp_ntz column, you get incorrect results:
{noformat}
set spark.sql.codegen.wholeStage=false;
select a, b, sum(c) from v1 group by a, b;
{noformat}
This query produces
{noformat}
2	2012-01-01 09:32:39.738368	20000
1	2013-01-01 09:32:39.738368	48000
2	2013-01-01 09:32:39.738368	30000
Time taken: 1.927 seconds, Fetched 4 row(s)
{noformat}