av9ash/GitBugs huggingface dataset:
Bug: Support star in groupBy.agg()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 12:00
Affects Version/s: 3.4.0

Doctest in {{pyspark.sql.connect.group.GroupedData.agg}} fails with the error below:

{code}
Failed example:
    df.groupBy(df.name).agg({"*": "count"}).sort("name").show()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.group.GroupedData.agg[4]>", line 1, in <module>
        df.groupBy(df.name).agg({"*": "count"}).sort("name").show()
      File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 538, in show
        print(self._show_string(n, truncate, vertical))
      File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 424, in _show_string
        pdf = DataFrame.withPlan(
      File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 895, in toPandas
        return self._session.client._to_pandas(query)
      File "/.../spark/python/pyspark/sql/connect/client.py", line 333, in _to_pandas
        return self._execute_and_fetch(req)
      File "/.../spark/python/pyspark/sql/connect/client.py", line 421, in _execute_and_fetch
        for b in self._stub.ExecutePlan(req, metadata=self._builder.metadata()):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/grpc/_channel.py", line 426, in __next__
        return self._next()
      File "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/grpc/_channel.py", line 826, in _next
        raise self
    grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    	status = StatusCode.UNKNOWN
    	details = "[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `*` cannot be resolved. Did you mean one of the following? [`age`, `name`];
    'Sort ['name DESC NULLS LAST], true
    +- 'Aggregate [name#26], [name#26, unresolvedalias('count('*), None)]
       +- Project [0#21L AS age#25L, 1#22 AS name#26]
          +- LocalRelation [0#21L, 1#22]
    "
    	debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:15002 {created_time:"2022-12-28T20:55:38.30791+09:00", grpc_status:2, grpc_message:"[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `*` cannot be resolved. Did you mean one of the following? [`age`, `name`];\n\'Sort [\'name DESC NULLS LAST], true\n+- \'Aggregate [name#26], [name#26, unresolvedalias(\'count(\'*), None)]\n   +- Project [0#21L AS age#25L, 1#22 AS name#26]\n      +- LocalRelation [0#21L, 1#22]\n"}"
{code}

We should enable this back after fixing the issue in Spark Connect
Bug: Remove *args: Any, **kwargs: Any for def transpose
Status: In Progress | Priority: Major | Resolution: 
Created: 31/Dec/22 11:59
Affects Version/s: 3.4.0

def transpose(self, *args: Any, **kwargs: Any) -> "Series":

This function only return self.copy()

[Sonar|https://sonarcloud.io/project/issues?languages=py&resolved=false&severities=BLOCKER&id=spark-python&open=AYQdnWmTRrJbVxW9ZCsf]
Bug: Window.partitionBy is not respected with row_number 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/22 13:05
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/window.py", line 292, in pyspark.sql.connect.window.Window.orderBy
Failed example:
    df.withColumn("row_number", row_number().over(window)).show()
Expected:
    +---+--------+----------+
    | id|category|row_number|
    +---+--------+----------+
    |  1|       a|         1|
    |  1|       a|         2|
    |  1|       b|         3|
    |  2|       a|         1|
    |  2|       b|         2|
    |  3|       b|         1|
    +---+--------+----------+
Got:
    +---+--------+----------+
    | id|category|row_number|
    +---+--------+----------+
    |  1|       b|         1|
    |  1|       a|         2|
    |  1|       a|         3|
    |  2|       b|         1|
    |  2|       a|         2|
    |  3|       b|         1|
    +---+--------+----------+
{code}
Bug: eqNullSafe does not support None as its argument
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/22 12:05
Affects Version/s: 3.4.0

{code}
**********************************************************************
File "/.../spark/python/pyspark/sql/connect/column.py", line 90, in pyspark.sql.connect.column.Column.eqNullSafe
Failed example:
    df1.select(
        df1['value'] == 'foo',
        df1['value'].eqNullSafe('foo'),
        df1['value'].eqNullSafe(None)
    ).show()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.column.Column.eqNullSafe[2]>", line 4, in <module>
        df1['value'].eqNullSafe(None)
      File "/.../workspace/forked/spark/python/pyspark/sql/connect/column.py", line 78, in wrapped
        return scalar_function(name, self, other)
      File "/.../workspace/forked/spark/python/pyspark/sql/connect/column.py", line 95, in scalar_function
        return Column(UnresolvedFunction(op, [arg._expr for arg in args]))
      File "/.../workspace/forked/spark/python/pyspark/sql/connect/column.py", line 95, in <listcomp>
        return Column(UnresolvedFunction(op, [arg._expr for arg in args]))
    AttributeError: 'NoneType' object has no attribute '_expr'
{code}
Bug: Support Window.rowsBetween
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 29/Dec/22 02:28
Affects Version/s: 3.4.0

Doctest in pyspark.sql.connect.window.Window.rowsBetween fails with the error below:
{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 324, in pyspark.sql.connect.column.Column.over
Failed example:
    window = Window.partitionBy("name").orderBy("age")                 .rowsBetween(Window.unboundedPreceding, Window.currentRow)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.column.Column.over[1]>", line 1, in <module>
        window = Window.partitionBy("name").orderBy("age")                 .rowsBetween(Window.unboundedPreceding, Window.currentRow)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/utils.py", line 346, in wrapped
        raise NotImplementedError()
    NotImplementedError{code}
We should enable this back after fixing the issue in Spark Connect
Bug: __getitem__ does not work with Column.isin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/22 12:09
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/column.py", line 364, in pyspark.sql.connect.column.Column.isin
Failed example:
    df[df.name.isin("Bob", "Mike")].collect()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.column.Column.isin[1]>", line 1, in <module>
        df[df.name.isin("Bob", "Mike")].collect()
      File "/.../workspace/forked/spark/python/pyspark/sql/connect/dataframe.py", line 888, in __getitem__
        return col(name)
      File "/.../workspace/forked/spark/python/pyspark/sql/connect/functions.py", line 161, in col
        return Column(ColumnReference(col))
      File "/.../spark/python/pyspark/sql/connect/expressions.py", line 322, in __init__
        assert isinstance(name, str)
    AssertionError
**********************************************************************
File "/.../spark/python/pyspark/sql/connect/column.py", line 366, in pyspark.sql.connect.column.Column.isin
Failed example:
    df[df.age.isin([1, 2, 3])].collect()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.column.Column.isin[2]>", line 1, in <module>
        df[df.age.isin([1, 2, 3])].collect()
      File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 888, in __getitem__
        return col(name)
      File "/.../spark/python/pyspark/sql/connect/functions.py", line 161, in col
        return Column(ColumnReference(col))
      File "/.../spark/python/pyspark/sql/connect/expressions.py", line 322, in __init__
        assert isinstance(name, str)
    AssertionError
{code}
Bug: Add the ability to create pvc before creating driver/executor pod
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Dec/22 03:20
Affects Version/s: 3.3.0

Creating pvc after driver/executor pod has Warning event from default-scheduler, such as
{code:java}
error getting PVC "spark/application-exec-1-pvc-0": could not find v1.PersistentVolumeClaim "spark/application-exec-1-pvc-0" {code}
Normal k8s workflow is to create PVC first and schedule pod to mount PVC.

 

We have a scenes that webhook server will try to reschedule pod and pvc to another pod. Because pvc creation after pod, wehbook couldn't find pvc based on pod metadata.
Bug: groupBy(...).agg(...).sort does not actually sort the output
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 12:02
Affects Version/s: 3.4.0

{code}
**********************************************************************
File "/.../spark/python/pyspark/sql/connect/group.py", line 211, in pyspark.sql.connect.group.GroupedData.agg
Failed example:
    df.groupBy(df.name).agg(F.min(df.age)).sort("name").show()
Differences (ndiff with -expected +actual):
      +-----+--------+
      | name|min(age)|
      +-----+--------+
    + |  Bob|       5|
      |Alice|       2|
    - |  Bob|       5|
      +-----+--------+
    + <BLANKLINE>
{code}
Bug: Remove OptimizedCreateHiveTableAsSelectCommand
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/22 05:47
Affects Version/s: 3.4.0

CTAS use a nested execution to do data writing, so it is unnecessary to have OptimizedCreateHiveTableAsSelectCommand. The inside InsertIntoHiveTable would be converted to InsertIntoHadoopFsRelationCommand if possible.
Bug: Fix arithmetic ops: negate, pow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/22 03:35
Affects Version/s: 3.4.0


pyspark.sql.connect.client.SparkConnectException: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "[UNRESOLVED_ROUTINE] Cannot resolve function `negate` on search path [`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`]."
        debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:15002 {grpc_message:"[UNRESOLVED_ROUTINE] Cannot resolve function `negate` on search path [`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].", grpc_status:2, created_time:"2022-12-29T11:27:23.72757+08:00"}"
Bug: Support Column.bitwiseAND,bitwiseOR,bitwiseXOR,eqNullSafe,isNotNull,isNull,isin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 13:20
Affects Version/s: 3.4.0

Doctest in pyspark.sql.connect.column.Column.bitwiseAnd fails with the error below:


{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 86, in pyspark.sql.connect.column.Column.bitwiseAND
Failed example:
    df.select(df.a.bitwiseAND(df.b)).collect()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.column.Column.bitwiseAND[2]>", line 1, in <module>
        df.select(df.a.bitwiseAND(df.b)).collect()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 896, in collect
        pdf = self.toPandas()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 910, in toPandas
        return self._session.client._to_pandas(query)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 337, in _to_pandas
        return self._execute_and_fetch(req)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 431, in _execute_and_fetch
        for b in self._stub.ExecutePlan(req, metadata=self._builder.metadata()):
      File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 426, in __next__
        return self._next()
      File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 826, in _next
        raise self
    grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    	status = StatusCode.UNKNOWN
    	details = "[UNRESOLVED_ROUTINE] Cannot resolve function `bitwiseAND` on search path [`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`]."
    	debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:15002 {grpc_message:"[UNRESOLVED_ROUTINE] Cannot resolve function `bitwiseAND` on search path [`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].", grpc_status:2, created_time:"2022-12-28T05:16:01.360735-08:00"}"
    >
{code}

We should enable this back after fixing the issue in Spark Connect
Bug: Support multiple arguments in groupBy.avg(...)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 12:16
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/group.py", line 188, in pyspark.sql.connect.group.GroupedData.avg
Failed example:
    df.groupBy().avg('age', 'height').show()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.group.GroupedData.avg[3]>", line 1, in <module>
        df.groupBy().avg('age', 'height').show()
    TypeError: avg() takes 2 positional arguments but 3 were given
{code}
Bug: Support multiple arguments in groupBy.sum(...)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 12:19
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/group.py", line 186, in pyspark.sql.connect.group.GroupedData.sum
Failed example:
    df.groupBy().sum("age", "height").show()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.group.GroupedData.sum[3]>", line 1, in <module>
        df.groupBy().sum("age", "height").show()
    TypeError: sum() takes 2 positional arguments but 3 were given
{code}
Bug: Support multiple arguments in groupBy.min(...)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 12:17
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/group.py", line 186, in pyspark.sql.connect.group.GroupedData.min
Failed example:
    df.groupBy().min("age", "height").show()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.group.GroupedData.min[3]>", line 1, in <module>
        df.groupBy().min("age", "height").show()
    TypeError: min() takes 2 positional arguments but 3 were given
{code}
Bug: Support multiple arguments in groupBy.max(...)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 12:07
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/group.py", line 181, in pyspark.sql.connect.group.GroupedData.max
Failed example:
    df.groupBy().max("age", "height").show()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.group.GroupedData.max[3]>", line 1, in <module>
        df.groupBy().max("age", "height").show()
    TypeError: max() takes 2 positional arguments but 3 were given
{code}
Bug: Session window: analysis rule "ResolveWindowTime" does not apply tree-pattern based pruning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/22 11:26
Affects Version/s: 3.4.0

We missed to apply tree-pattern based pruning in the rule ResolveWindowTime, which leads to evaluate ResolveWindowTime unnecessarily to the other logical nodes multiple times.
Bug: Implement `functions.format_number`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/22 22:49
Affects Version/s: 3.4.0

Currently, `functions.format_number` has data type mismatch issue as shown below:

{code:python}
>>> from pyspark.sql.connect.session import SparkSession as RemoteSparkSession
>>> cspark = RemoteSparkSession.builder.remote().getOrCreate()
>>> from pyspark.sql.connect import functions as CF
>>> cspark.createDataFrame([(5,)], ['a']).select(CF.format_number('a', 4))
22/12/09 14:46:52 ERROR SparkConnectService: Error analyzing plan.
org.apache.spark.sql.AnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "format_number(a, 4)" due to data type mismatch: Parameter 2 requires the ("INT" or "STRING") type, however "4" has the type "BIGINT".;
'Project [unresolvedalias(format_number(a#2L, 4), None)]
+- Project [0#0L AS a#2L]
   +- LocalRelation [0#0L]
...
{code}

The jira aims to fix the data type mismatch issue and implement `functions.format_number`.
Bug: Introduce AgnosticEncoders
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/22 18:19
Affects Version/s: 3.4.0

For the Spark Connect Scala Client we need Encoders. We cannot use the current ExpressionEncoder infrastructure because of the following reasons:
 * Connect uses Arrow instead of internal rows
 * We don't want to have a dependency on Catalyst in the client.
 * We don't want to be sending the expressions that expression encoders use over the wire.

An additional challenge is that we cannot send TypeTags over the wire.

The proposal is to create an intermediate encoder. This encoder describes exactly how which external type maps to an internal type, but it does not prescribe which format is used.

An upside of this approach is that we can reduce the cost of encoder creation by quite a margin.
Bug: Support parameterized SQL in PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/22 13:55
Affects Version/s: 3.4.0

Enhance the PySpark SQL API with support for parameterized SQL statements to improve security and reusability. Application developers will be able to write SQL with parameter markers whose values will be passed separately from the SQL code and interpreted as literals. This will help prevent SQL injection attacks for applications that generate SQL based on a user’s selections, which is often done via a user interface.

PySpark has already supported formatting of sqlText using the syntax {...}. Need to leave the API the same:

{code:python}
def sql(self, sqlQuery: str, **kwargs: Any) -> DataFrame:
{code}

and support new parameters by the same API.

PySpark *sql()* should passes unused parameters to the JVM side where the Java sql() method handles them. For example:

{code:python}
>>> mydf = spark.range(10)
>>> spark.sql("SELECT id FROM {mydf} WHERE id % @param1 = 0", mydf=mydf, param1='3').show()
+---+
| id|
+---+
|  0|
|  3|
|  6|
|  9|
+---+
{code}
Bug: ORDER BY ALL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/22 00:35
Affects Version/s: 3.3.0

This patch adds ORDER BY ALL support to SQL. ORDER BY ALL is a syntactic sugar to sort the output by all the fields, from left to right. It also allows specifying asc/desc as well as null ordering. This was initially introduced by DuckDB.
Bug: Introduce builtin and session namespaces for builtin functions and temp views/functions 
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Dec/22 21:32
Affects Version/s: 3.4.0

Spark today allows overloading between persisted relations and function and temporary relations and functions.
It also allows overloading between persisted functions and builtin functions.

While Spark allows us to disambiguate a persisted objects by qualifying it, there is no qualifier for temp or builtin objects.

Here we propose to use `builtin` for builtin objects and `session` for session temporary objects.
If there is a conflict with persisted schemas of this name we can further declare that the catalog for both is `system`
Bug: Setup initial client module for Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/22 15:47
Affects Version/s: 3.3.2

In https://issues.apache.org/jira/browse/SPARK-41369, the connect module was split into server/common to extract dependencies for the Scala client. 
With this extraction completed, the client module can be setup in preparation for the Scala client.
Bug: DECODE function returns wrong results when passed NULL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/22 19:09
Affects Version/s: 3.2.0

The DECODE function was implemented for Oracle compatibility. It works similar to CASE expression, but it is supposed to have one major difference: NULL == NULL
[https://docs.oracle.com/database/121/SQLRF/functions057.htm#SQLRF00631]

The Spark implementation does not observe this however:

select decode(null, 6, 'Spark', NULL, 'SQL', 4, 'rocks');

NULL

 The result is supposed to be 'SQL'
Bug: json expressions much slower in optimized mode
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/22 07:14
Affects Version/s: 3.1.3

I noticed spark structured streaming reading from Kafka json string into struct type is much slower in spark-3.1+ than spark-3.0. Profiling reveals the json expressions in spark-3.0 mostly on evaluate subExpr, while spark-3.1/3.2 spent a lot time on writeField. 

Suspect this may be related to SPARK-32948, so I tried with add a bogus option 

from_json($"value", mySchema, Map("bogus_key"-> "bogus_value")

this turns off the optimization and the performance is much better. For reference, 

for same amount #records, it is 30 seconds vs. 3 minute on a task processing 500k records. This is big difference for a streaming job.
Bug: Spark streaming query scheduling synchronisation  with Trigger Interval
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/22 12:57
Affects Version/s: 2.4.8

Hi,
We detect a strange behavior on spark streaming when we set a trigger interval for example at 1 minutes all query will start at 0:00:00 0:01:00 0:02:00 no matter the start time of the query.
So all query are "sync", so it's can disturbed a cluster a cluster i do leads to spike of utilisation 
!image-2022-12-21-07-57-18-679.png!

For me the expected behavior should be like this

!image-2022-12-21-07-57-32-654.png!

 

It's because of this line 
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala#L98]

as now in intervalMS are long now / intervalMs * intervalMs will just cut in my case the second, as it's explicitely like this on the test ([https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ProcessingTimeExecutorSuite.scala#L36)]  i do not know if it's the expected behavior or it's juste because this line it's here since 6 years. So it's affecting all versions since 6 years. 

Regards

Thomas