av9ash/GitBugs huggingface dataset:
Bug: Incorrect parsing of negative ANSI typed interval literals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/21 09:21
Affects Version/s: 3.2.0

If start field and end field are the same, parser doesn't take into account the sign before interval literal string. For example:

Works fine:
{code:sql}
spark-sql> select interval -'1-1' year to month;
-1-1
{code}

Incorrect result:
{code:sql}
spark-sql> select interval -'1' year;
1-0
{code}
Bug: Stop task result getter properly on spark context stopping
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Sep/21 10:51
Affects Version/s: 3.3.0

org.apache.spark.scheduler.TaskSchedulerImpl#stop doesn't handle exception properly. If one component throws exceptions on stopping, the exception is thrown and TaskSchedulerImpl.stop() will not be executed completely.

For example if backend.stop() fails, then taskResultGetter.stop() won't be executed. The result is that after a couple of restart of the spark context, there will be a lot of '

task-result-getter' threads retained.

 

!image-2021-09-24-18-50-57-072.png!

!image-2021-09-24-18-51-03-837.png!
Bug: CollapseProject should not collapse correlated scalar subqueries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/21 00:01
Affects Version/s: 3.3.0

Currently, the optimizer rule `CollapseProject` inlines expressions generated from correlated scalar subqueries, which can create unnecessary left outer joins.

{code:sql}
select c1, s, s * 10 from (
select c1, (select first(c2) from t2 where t1.c1 = t2.c1) s from t1)
{code}

{code:scala}
// Before
Project [c1, s, (s * 10)]
+- Project [c1, scalar-subquery [c1] AS s]
   :  +- Aggregate [c1], [first(c2), c1] 
   :      +- LocalRelation [c1, c2]
   +- LocalRelation [c1, c2]

// After (scalar subqueries are inlined)
Project [c1, scalar-subquery [c1], (scalar-subquery [c1] * 10)]
:  +- Aggregate [c1], [first(c2), c1] 
:      +- LocalRelation [c1, c2]
:  +- Aggregate [c1], [first(c2), c1] 
:      +- LocalRelation [c1, c2]
+- LocalRelation [c1, c2]
{code}

Then this query will have two LeftOuter joins created. We should only collapse projects after correlated subqueries are rewritten as joins.
Bug: Do not collapse Project with Aggregate when correlated subqueries are present in the project list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/21 20:25
Affects Version/s: 3.2.0

Currently CollapseProject combines Project with Aggregate when the shared attributes are deterministic. But if there are correlated scalar subqueries in the project list that uses the output of the aggregate, they cannot be combined. Otherwise, the plan after rewrite will not be valid:

{code}
select (select sum(c2) from t where c1 = cast(s as int)) from (select sum(c2) s from t)

== Optimized Logical Plan ==
Aggregate [sum(c2)#10L AS scalarsubquery(s)#11L]
+- Project [sum(c2)#10L]
   +- Join LeftOuter, (c1#2 = cast(sum(c2#3) as int))
      :- LocalRelation [c2#3]
      +- Aggregate [c1#2], [sum(c2#3) AS sum(c2)#10L, c1#2]
         +- LocalRelation [c1#2, c2#3]

java.lang.UnsupportedOperationException: Cannot generate code for expression: sum(input[0, int, false])
{code}
Bug: Create shaded Hive module and upgrade to higher version of Guava
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Sep/21 16:33
Affects Version/s: 3.3.0

Currently Spark is tied with Guava from Hive which is of version 14. This proposes to create a separate module {{hive-shaded}} which shades dependencies from Hive and subsequently allows us to upgrade Guava independently.
Bug: array_union should not produce duplicated result
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/21 12:36
Affects Version/s: 2.4.0

{code:java}
select array_union(array(cast('nan' as double), cast('nan' as double)), array())
{code}
This returns [NaN, NaN], but it should return [NaN]
Bug: BroadcastNestedLoopJoinExec should use all condition instead of non-equi condition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/21 01:57
Affects Version/s: 3.2.0

At JoinSelection, with ExtractEquiJoinKeys, we use `nonEquiCond` as the join condition. It's wrong since there should exist some equi condition.

{code:java}
Seq(joins.BroadcastNestedLoopJoinExec(
  planLater(left), planLater(right), buildSide, joinType, nonEquiCond))
{code}

But it's should not be a bug, since we always use the smj as the default join strategy for ExtractEquiJoinKeys.
Bug: Support broadcast nested loop join hint for equi-join
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Sep/21 04:59
Affects Version/s: 3.3.0

For the join if one side is small and other side is large, the shuffle overhead is also very big. Due to the 
bhj limitation, we can only broadcast right side for left join and left side for right join. So for the other case, we can try to use `BroadcastNestedLoopJoin` as the join strategy.
Bug: Make class ColumnarBatch extendable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/21 01:17
Affects Version/s: 3.3.0

To support better vectorized reading in multiple data source, ColumnarBatch need to be extendable. For example, To support row-level delete(  [https://github.com/apache/iceberg/issues/3141]) in Iceberg's vectorized read, we need to filter out deleted rows in a batch, which requires ColumnarBatch to be extendable.
Bug: Fix ps.DataFrame.isin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/21 18:11
Affects Version/s: 3.3.0

{code:python}
>>> psdf = ps.DataFrame(
...     {"a": [None, 2, 3, 4, 5, 6, 7, 8, None], "b": [None, 5, None, 3, 2, 1, None, 0, 0], "c": [1, 5, 1, 3, 2, 1, 1, 0, 0]},
... )
>>> 
>>> psdf
     a    b  c                                                                  
0  NaN  NaN  1
1  2.0  5.0  5
2  3.0  NaN  1
3  4.0  3.0  3
4  5.0  2.0  2
5  6.0  1.0  1
6  7.0  NaN  1
7  8.0  0.0  0
8  NaN  0.0  0
>>> other = [1, 2, None]

>>> psdf.isin(other)
      a     b     c
0  None  None  True
1  True  None  None
2  None  None  True
3  None  None  None
4  None  True  True
5  None  True  True
6  None  None  True
7  None  None  None
8  None  None  None
>>> psdf.isin(other).dtypes
a    bool
b    bool
c    bool
dtype: object
>>> psdf.to_pandas().isin(other).dtypes
a    bool
b    bool
c    bool
dtype: object
>>> psdf.to_pandas().isin(other)
       a      b      c
0  False  False   True
1   True  False  False
2  False  False   True
3  False  False  False
4  False   True   True
5  False   True   True
6  False  False   True
7  False  False  False
8  False  False  False
>>> 

{code}
Bug: Kryo Buffer underflow error in Spark 3.1
Status: Open | Priority: Major | Resolution: 
Created: 17/Sep/21 00:51
Affects Version/s: 3.1.2

Spark JavaPairRDD processing fails with error

{code:java}
com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
topologyInfo_ (org.apache.spark.storage.BlockManagerId)
loc (org.apache.spark.scheduler.HighlyCompressedMapStatus)
	at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
	at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:373)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:127)
	at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:804)
	at com.twitter.chill.TraversableSerializer.read(Traversable.scala:43)
	at com.twitter.chill.TraversableSerializer.read(Traversable.scala:21)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:734)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:734)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:816)
	at org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:397)
	at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:103)
	at org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1(TaskResultGetter.scala:75)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
	at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:63)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{code}

Kryo log shows this right before the error, could this be a mulithreading issue in spark!
{code}
0:15 TRACE: [kryo] Read field: loc (org.apache.spark.scheduler.HighlyCompressedMapStatus) pos=534
00:15 TRACE: [kryo] Read field: loc (org.apache.spark.scheduler.HighlyCompressedMapStatus) pos=726
00:15 TRACE: [kryo] Read class 15: org.apache.spark.storage.BlockManagerId
00:15 TRACE: [kryo] Read class 15: org.apache.spark.storage.BlockManagerId
00:15 TRACE: [kryo] Read field: topologyInfo_ (org.apache.spark.storage.BlockManagerId) pos=801
00:15 TRACE: [kryo] Read field: topologyInfo_ (org.apache.spark.storage.BlockManagerId) pos=609
{code}
Bug: Fix Series.isin when Series has NaN values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/21 05:08
Affects Version/s: 3.2.0

{code:python}
>>> pser = pd.Series([None, 5, None, 3, 2, 1, None, 0, 0])
>>> psser = ps.from_pandas(pser)
>>> pser.isin([1, 3, 5, None])
0    False
1     True
2    False
3     True
4    False
5     True
6    False
7    False
8    False
dtype: bool
>>> psser.isin([1, 3, 5, None])
0    None                                                                       
1    True
2    None
3    True
4    None
5    True
6    None
7    None
8    None
dtype: object

{code}
Bug: Make dev/mima runs on Java 17
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/21 10:11
Affects Version/s: 3.3.0

It will throw {{InaccessibleObjectException}} if run {{dev/mima}} on Java 17.
{noformat}
Exception in thread "main" java.lang.reflect.InaccessibleObjectException: Unable to make public java.lang.String java.util.jar.JarFile$JarFileEntry.getName() accessible: module java.base does not "opens java.util.jar" to unnamed module @755c9148
{noformat}
Bug: Fix race-condition on "ensure continuous stream is being used" in KafkaContinuousTest
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/21 06:47
Affects Version/s: 2.4.8

The test “ensure continuous stream is being used“ in KafkaContinuousTestquickly checks the actual type of the execution, and stop the query. Stopping the streaming query in continuous mode is done by interrupting query execution thread and join indefinitely.

In parallel, started streaming query is going to generate execution plan, including running optimizer. Some parts of SessionState can be built at that time, as they are defined as lazy. The problem is, some of them seem to be able to “swallow” the InterruptedException and let the thread run continuously.

That said, the query can’t indicate whether there is a request on stopping query, so the query won’t stop.
Bug: Improve run-time performance for window function first and last against UnboundedFollowing window ROWS frame
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Sep/21 19:08
Affects Version/s: 2.4.0

*Context*

The *UnboundedFollowingWindowFunctionFrame* has the time complexity of *O(N^2)*, N is the number of rows in the current partition, more specific the complexity is O(N* (N - 1)/2).

What happens internally in UnboundedFollowingWindowFunctionFrame:
 In the window frame, while processing each incoming row, it will go through current row till the end of partition to do re-calculation. This process will be repeated on each incoming row, which causes the high run-time complexity.

But UnboundedPrecedingWindowFunctionFrame has much better time complexity O(N), N is the number of rows in the current partition.

 

*What is the idea of the improvement?*

Give the big time complexity difference between UnboundedFollowingWindowFunctionFrame and UnboundedPrecedingWindowFunctionFrame, we can do following conversions to improve the time complexity of first() and last() from O(N^2) to O(N)
{code:java}
case 1:
first() OVER(PARTITION BY colA ORDER BY colB ASC ROWS CURRENT ROW AND UNBOUNDED FOLLOWING) 
converts to 
last()  OVER(PARTITION BY colA ORDER BY colB DEAC ROWS UNBOUNDED PRECEDING AND CURRENT ROW)

case 2:
last()  OVER(PARTITION BY colA ORDER BY colB ASC ROWS CURRENT ROW AND UNBOUNDED FOLLOWING) 
converts to 
first() OVER(PARTITION BY colA ORDER BY colB DESC ROWS UNBOUNDED PRECEDING AND CURRENT ROW)

{code}
 

*Summary*

Replace "UNBOUNDED FOLLOWING" with "UNBOUNDED PRECEDING", and flip the ORDER BY for the window functions first() and last() for ROWS.
Bug: Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 15:56
Affects Version/s: 3.3.0

To make it easy to use/modify the semantic annotation, we want to have a shorter API to update the metadata in a dataframe.

Currently we have

{code:scala}
df.withColumn("col1", col("col1").alias("col1", metadata=metadata))
{code}

to update the metadata without changing the column name, and this is too verbose. We want to have a syntax suger API

{code:scala}
df.withMetadata("col1", metadata=metadata)
{code}

to achieve the same functionality.

A bit of background for the frequency of the metadata update: We are working on inferring the semantic data types and use them in AutoML and store the semantic annotation in the metadata. So in many cases, we will suggest the user update the metadata to correct the wrong inference or manually add the annotation for weak inference.
Bug: Adjust overhead of cached relation for DPP
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/21 05:33
Affects Version/s: 3.3.0

Currently we calculate if there is benefit of pruning with DPP by simply summing up the size of all scan relations as the overhead. However, for cached relations, the overhead should be different than a non-cached relation. This proposes to use adjusted overhead for cached relation with DPP.
Bug: Problems with update function in koalas - pyspark pandas.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Sep/21 10:39
Affects Version/s: 3.2.0

Hi I am using "from pyspark import pandas as ps" in a master build yesterday. 
I do have some columns that I need to join to one. 
In pandas I use update.


54 FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION 23 non-null object 
55 FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION.P 24348 non-null object
 
 
 pd1['FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION'].update(pd1['FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION.P'])
 
 ---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
/tmp/ipykernel_73/391781247.py in <module>
----> 1 pd1['FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION'].update(pd1['FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION.P'])

/opt/spark/python/pyspark/pandas/series.py in update(self, other)
 4549 raise TypeError("'other' must be a Series")
 4550 
-> 4551 combined = combine_frames(self._psdf, other._psdf, how="leftouter")
 4552 
 4553 this_scol = combined["this"]._internal.spark_column_for(self._column_label)

/opt/spark/python/pyspark/pandas/utils.py in combine_frames(this, how, preserve_order_column, *args)
 139 elif len(args) == 1 and isinstance(args[0], DataFrame):
 140 assert isinstance(args[0], DataFrame)
--> 141 assert not same_anchor(
 142 this, args[0]
 143 ), "We don't need to combine. `this` and `that` are same."

AssertionError: We don't need to combine. `this` and `that` are same.


pd1.info()

54 FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION 23 non-null object 
55 FD_OBJECT_SUPPLIES_SERVICES_OBJECT_SUPPLY_SERVICE_ADDITIONAL_INFORMATION.P 24348 non-null object
Bug: SQL column nullable setting not retained as part of spark read
Status: Open | Priority: Major | Resolution: 
Created: 14/Sep/21 20:52
Affects Version/s: 3.1.1

When reading in a column set as not null this is not retained as part of the spark.read.
All columns are showing as nullable = true

Is there a way to change this behaviour to retain the null setting from the source?

See here for more info

https://github.com/microsoft/sql-spark-connector/issues/121

 

Example code from databricks:

tableName = "dbo.MyTable"

df = spark.read
.format("com.microsoft.sqlserver.jdbc.spark")
.option("url", myJdbcUrl)
.option("accessToken", accessToken)
.option("dbTable", tableName)
.load()

df.printSchema()
Bug: Support ILIKE - case insensitive Like
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/21 12:38
Affects Version/s: 3.3.0

Add ILIKE (case insensitive LIKE) to improve user experience with Spark SQL. Other DBMSs have the feature:
- PostgreSQL https://www.postgresql.org/docs/12/functions-matching.html
- Snowflake https://docs.snowflake.com/en/sql-reference/functions/ilike.html#ilike
Bug: Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/21 07:10
Affects Version/s: 3.3.0

Add ILIKE (case insensitive LIKE) to improve user experience with Spark SQL. Other DBMSs have the feature:
- Snowflake https://docs.snowflake.com/en/sql-reference/functions/ilike_any.html
- PostgreSQL https://www.postgresql.org/docs/12/functions-matching.html
Bug: InaccessibleObjectException in Java 9+ on startup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/21 16:37
Affects Version/s: 3.2.0

A user reported this error on startup, which must be Java9+-related given the Java-9+ exception:

{code}
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @71e9ddb4
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
        at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
        at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:56)
{code}

Code in this section tries to open up some access to DirectBuffer by reflection, which no longer works by default in Java 9+. We long ago tried to handle this by dealing with the exception and avoiding reflection where not possible. However this seems like a possible new way the same type of issue can manifest.

We can shore up the code checking this to try to handle this apparent error path similarly.
Bug: day-time interval types should respect daylight saving time correctly
Status: Open | Priority: Major | Resolution: 
Created: 11/Sep/21 13:10
Affects Version/s: 3.2.0

In the current master, day-time interval types handle 24 hours as 1 day regardless of the time zone.
So, some operation with day-time interval data doesn't respect daylight saving time.
{code}
spark-sql> SET spark.sql.legacy.interval.enabled=false;
spark-sql> SET spark.sql.session.timeZone America/Los_Angeles

spark-sql> SELECT timestamp'2019-03-10 01:00:00' + INTERVAL '1' DAY;
2019-03-11 01:00:00 -- OK. Expected result.

spark-sql> SELECT timestamp'2019-03-10 01:00:00' + INTERVAL '24' HOUR;
2019-03-11 01:00:00 -- Not OK. 2019-03-11 02:00:00 is expected.
{code}

On the other hand, non-ANSI interval types properly handle daylight saving time.
{code}
spark-sql> SET spark.sql.legacy.interval.enabled=true;
spark-sql> SET spark.sql.session.timeZone America/Los_Angeles

spark-sql> SELECT timestamp'2019-03-10 01:00:00' + INTERVAL '1' DAY;
2019-03-11 01:00:00

spark-sql> SELECT timestamp'2019-03-10 01:00:00' + INTERVAL '24' HOUR;
2019-03-11 02:00:00
{code}
Bug: Support Series.__and__ for Integral
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Sep/21 02:49
Affects Version/s: 3.3.0

{code:python}
>>> pser1 = pd.Series([1, 2, 3])
>>> pser2 = pd.Series([4, 5, 6, 7])
>>> pser1 & pser2
0    False
1    False
2     True
3    False
dtype: bool

>>> pser1 = pd.Series([1, 2, 3])
>>> pser2 = pd.Series([4, 5, 6])
>>> pser1 & pser2
0    0
1    0
2    2
dtype: int64

>>> pser1 = ps.Series([1, 2, 3])
>>> pser2 = ps.Series([4, 5, 6, 7])
>>> pser1 & pser2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dgd/spark/python/pyspark/pandas/base.py", line 423, in __and__
    return self._dtype_op.__and__(self, other)
  File "/Users/dgd/spark/python/pyspark/pandas/data_type_ops/base.py", line 317, in __and__
    raise TypeError("Bitwise and can not be applied to %s." % self.pretty_name)
TypeError: Bitwise and can not be applied to integrals.

{code}
Bug: Implement spark-shell idle timeouts
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Sep/21 08:27
Affects Version/s: 3.1.2

Many customers have been asking if there is a setting they can use to kill idle spark-shell since they can't really go to each developers desk and force them to use Cntr+D or exit() when their work is over. Our response so far has been to use dynamic allocation so that it will release the executors after the specified timeout.

However this is not always an ideal solution since the shell process would still be there, though AM would be occupying a very small resource and the user still needs to kill the idle spark shell via CM> Applications > spark-shell > Kill or run 'kill -9' on the OS to remove those. It would be nice to have a property in Spark (and exposed in CM) which deals with idle spark-shells, just like we have in beeline and let's leave it to the admins to see if they want the idle spark-shell timeout to be set as 1 day or a week.
Bug: Close resources properly in StateStoreSuite/RocksDBStateStoreSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Sep/21 03:14
Affects Version/s: 3.2.0

The StateStoreProvider instances created from "newStoreProvider" are NOT automatically closed.

While this is trivial for HDFSBackedStateStoreProvider, for RocksDBStateStoreProvider we leak RocksDB instance as well which should have closed. Most tests in the RocksDBStateStoreSuite initialize RocksDBStateStoreProvider, meaning that 60+ RocksDB instances are not closed in the suite.
Bug: SQL sequence function with interval returns unexpected error in latest versions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/21 12:17
Affects Version/s: 3.0.3

For example this returns {color:#FF0000}java.lang.ArrayIndexOutOfBoundsException: 1 {color}
{code:java}
select sequence(
 date_trunc('month', '2021-08-30'),
 date_trunc('month', '2021-08-15'),
 - interval 1 month){code}
Another cases like - all ok
{code:java}
select sequence(
 date_trunc('month', '2021-07-15'),
 date_trunc('month', '2021-08-30'),
 interval 1 month) as x

 , sequence(
 date_trunc('month', '2021-08-30'),
 date_trunc('month', '2021-07-15'),
 - interval 1 month) as y

 , sequence(
 date_trunc('month', '2021-08-15'),
 date_trunc('month', '2021-08-30'),
 interval 1 month) as z{code}

In version 3.0.0 this works