av9ash/GitBugs huggingface dataset:
Bug: Introduce UpdatingSessionIterator adjusting session information on elements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 03:09
Affects Version/s: 3.2.0

This issue tracks effort on introducing UpdatingSessionIterator, which analyzes neighbor elements and adjust session information on elements.

This issue also introduces UpdatingSessionExec which is the physical node on leveraging UpdatingSessionIterator to sort the input rows and updates session information on input rows.
Bug: Port/integrate Koalas DataFrame unit test into PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 02:52
Affects Version/s: 3.2.0

This JIRA aims to port [Koalas DataFrame test|https://github.com/databricks/koalas/tree/master/databricks/koalas/tests/test_dataframe.py] appropriately to [PySpark tests|https://github.com/apache/spark/tree/master/python/pyspark/tests].
Bug: Whether to expose outputDeterministicLevel so custom RDDs can set deterministic level
Status: Open | Priority: Major | Resolution: 
Created: 04/Mar/21 01:16
Affects Version/s: 3.2.0

This ticket is open to track a TODO item in RDD.outputDeterministicLevel.

We need to decide if we want to expose it so users can set deterministic level to their custom RDDs.
Bug: ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/21 17:42
Affects Version/s: 3.0.0

The current implementation of ExecutoMetricsPoller uses task count in each stage to decide whether to keep a stage entry or not. In the case of the executor only has 1 core, it may have these issues:
 # Peak metrics missing (due to stage entry being removed within a heartbeat interval)
 # Unnecessary and frequent hashmap entry removal and insertion.

Assuming an executor with 1 core has 2 tasks (task1 and task2, both belong to stage (0,0)) to execute in a heartbeat interval, the workflow in current ExecutorMetricsPoller implementation would be:

1. task1 start -> stage (0, 0) entry created in stageTCMP, task count increment to1

2. 1st poll() -> update peak metrics of stage (0, 0)

3. task1 end -> stage (0, 0) task count decrement to 0, stage (0, 0) entry removed, peak metrics lost.

4. task2 start -> stage (0, 0) entry created in stageTCMP, task count increment to1

5. 2nd poll() -> update peak metrics of stage (0, 0)

6. task2 end -> stage (0, 0) task count decrement to 0, stage (0, 0) entry removed, peak metrics lost

7. heartbeat() ->  empty or inaccurate peak metrics for stage(0,0) reported.

We can fix the issue by keeping entries with task count = 0 in stageTCMP map until a heartbeat occurs. At the heartbeat, after reporting the peak metrics for each stage, we scan each stage in stageTCMP and remove entries with task count = 0.

After the fix, the workflow would be:

1. task1 start -> stage (0, 0) entry created in stageTCMP, task count increment to1

2. 1st poll() -> update peak metrics of stage (0, 0)

3. task1 end -> stage (0, 0) task count decrement to 0,but the entry (0,0) still remain.

4. task2 start -> task count of stage (0,0) increment to1

5. 2nd poll() -> update peak metrics of stage (0, 0)

6. task2 end -> stage (0, 0) task count decrement to 0,but the entry (0,0) still remain.

7. heartbeat() ->  accurate peak metrics for stage (0, 0) reported. Remove entry for stage (0,0) in stageTCMP because its task count is 0.

 

How to verify the behavior? 

Submit a job with a custom polling interval (e.g., 2s) and spark.executor.cores=1 and check the debug logs of ExecutoMetricsPoller.
Bug: Support DPP in AQE when the broadcast exchange can be reused
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/21 07:41
Affects Version/s: 3.2.0

We have supported DPP in AQE when the join is Broadcast hash join before applying the AQE rules in SPARK-34168, which has some limitations. It only apply DPP when the small table side executed firstly and then the big table side can reuse the broadcast exchange in small table side. This Jira is to address the above limitations and can apply the DPP when the broadcast exchange can be reused.
Bug: Difference in results of casting float to timestamp
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/21 12:12
Affects Version/s: 3.2.0

The code below portraits the issue:
{code:sql}
spark-sql> CREATE TEMP VIEW v1 AS SELECT 16777215.0f AS f;
spark-sql> SELECT * FROM v1;
1.6777215E7
spark-sql> SELECT CAST(f AS TIMESTAMP) FROM v1;
1970-07-14 07:20:15
spark-sql> CACHE TABLE v1;
spark-sql> SELECT * FROM v1;
1.6777215E7
spark-sql> SELECT CAST(f AS TIMESTAMP) FROM v1;
1970-07-14 07:20:14.951424
{code}

The result from the cached view *1970-07-14 07:20:14.951424* is different from un-cached view *1970-07-14 07:20:15*.
Bug: Spark UI not showing memory used of partitions in memory
Status: Open | Priority: Major | Resolution: 
Created: 13/Mar/21 16:34
Affects Version/s: 2.4.4

We have a job that caches RDDs into memory.  We know the code to cache is working as the spark logs correctly point out the caching is happening:

```
21/03/13 16:17:38 INFO BlockManagerInfo: *Added rdd_12_413 in memory* on ip-172-31-24-152.eu-west-2.compute.internal:43849 *(size: 201.4 MB, free: 575.0 GB)*
21/03/13 16:17:38 INFO TaskSetManager: Starting task 897.0 in stage 1.0 (TID 10277, ip-172-31-24-152.eu-west-2.compute.internal, executor 8, partition 897, RACK_LOCAL, 9812 bytes)
21/03/13 16:17:38 INFO TaskSetManager: Finished task 413.0 in stage 1.0 (TID 9793) in 1250463 ms on ip-172-31-24-152.eu-west-2.compute.internal (executor 8) (34/2162)
21/03/13 16:17:42 INFO BlockManagerInfo: Added rdd_12_768 in memory on ip-172-31-23-154.eu-west-2.compute.internal:37957 (size: 718.4 MB, free: 574.6 GB)
21/03/13 16:17:43 INFO TaskSetManager: Starting task 898.0 in stage 1.0 (TID 10278, ip-172-31-23-154.eu-west-2.compute.internal, executor 7, partition 898, RACK_LOCAL, 9841 bytes)
21/03/13 16:17:43 INFO TaskSetManager: Finished task 768.0 in stage 1.0 (TID 10148) in 1254945 ms on ip-172-31-23-154.eu-west-2.compute.internal (executor 7) (35/2162)
21/03/13 16:18:44 INFO BlockManagerInfo: Added rdd_12_409 in memory on ip-172-31-21-66.eu-west-2.compute.internal:38921 (size: 177.6 MB, free: 575.1 GB)
21/03/13 16:18:45 INFO TaskSetManager: Starting task 899.0 in stage 1.0 (TID 10279, ip-172-31-21-66.eu-west-2.compute.internal, executor 4, partition 899, RACK_LOCAL, 9828 bytes)
21/03/13 16:18:45 INFO TaskSetManager: Finished task 409.0 in stage 1.0 (TID 9789) in 1316584 ms on ip-172-31-21-66.eu-west-2.compute.internal (executor 4) (36/2162)
21/03/13 16:19:40 INFO BlockManagerInfo: Added rdd_12_400 in memory on ip-172-31-21-66.eu-west-2.compute.internal:38921 (size: 187.9 MB, free: 574.9 GB)
21/03/13 16:19:41 INFO TaskSetManager: Starting task 900.0 in stage 1.0 (TID 10280, ip-172-31-21-66.eu-west-2.compute.internal, executor 4, partition 900, RACK_LOCAL, 9843 bytes)
21/03/13 16:19:41 INFO TaskSetManager: Finished task 400.0 in stage 1.0 (TID 9780) in 1372717 ms on ip-172-31-21-66.eu-west-2.compute.internal (executor 4) (37/2162)
21/03/13 16:20:55 INFO BlockManagerInfo: Added rdd_12_640 in memory on ip-172-31-17-157.eu-west-2.compute.internal:34005 (size: 576.1 MB, free: 574.7 GB)
21/03/13 16:20:58 INFO TaskSetManager: Starting task 901.0 in stage 1.0 (TID 10281, ip-172-31-17-157.eu-west-2.compute.internal, executor 9, partition 901, RACK_LOCAL, 9750 bytes)
21/03/13 16:20:58 INFO TaskSetManager: Finished task 640.0 in stage 1.0 (TID 10020) in 1449618 ms on ip-172-31-17-157.eu-west-2.compute.internal (executor 9) (38/2162)
21/03/13 16:21:07 INFO BlockManagerInfo: Added rdd_12_610 in memory on ip-172-31-30-188.eu-west-2.compute.internal:38111 (size: 582.2 MB, free: 574.7 GB)
```

But when we look in the Spark UI Executors tab it shows 0 B used of the maximum. Please see screenshot:


 !Screenshot 2021-03-13 at 16.31.06.png!
Bug: collect_list(struct()) fails when used with GROUP BY
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/21 07:58
Affects Version/s: 3.1.1

The following is failing in DBR8.0 / Spark 3.1.1, but works in earlier DBR and Spark versions:
{quote}with step_1 as (
    select 'E' as name, named_struct('subfield', 1) as field_1
)
select name, collect_list(struct(field_1.subfield))
from step_1
group by 1
{quote}
Fails with the following error message:
{quote}AnalysisException: cannot resolve 'struct(step_1.`field_1`.`subfield`)' due to data type mismatch: Only foldable string expressions are allowed to appear at odd position, got: NamePlaceholder
{quote}
If you modify the query in any of the following ways then it still works::
 * if you remove the field "name" and the "group by 1" part of the query
 * if you remove the "struct()" from within the collect_list()
 * if you use "named_struct()" instead of "struct()" within the collect_list()

Similarly collect_set() is broken and possibly more related functions, but I haven't done thorough testing.
Bug: Discrepancy between TIMESTAMP_SECONDS and cast from float
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/21 07:46
Affects Version/s: 3.2.0

TIMESTAMP_SECONDS and CAST(<float> AS TIMESTAMP) have different results:
{code:sql}
spark-sql> SELECT CAST(16777215.0f AS TIMESTAMP);
1970-07-14 07:20:15
spark-sql> SELECT TIMESTAMP_SECONDS(16777215.0f);
1970-07-14 07:20:14.951424
{code}
Bug: Support int64 buffer lengths in Java for pyspark Pandas UDF as buffer expanding
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 02/Mar/21 06:12
Affects Version/s: 3.0.2

This issue is an extention of [arrow issue|https://issues.apache.org/jira/browse/ARROW-10957#] for making possible using pyspark Pandas UDF functions for data more than 2gb per data group.

Here is the deal - arrow [supports |https://github.com/apache/arrow/commit/9742007c463e253e2b916e65f668146953456a00#diff-2e086b32ec292aae20695dd4341c647c9a9d7d3d77816bf849f7fbf68e9fa6cfR209]long type for data serialization between java and python but spark doesn't. It gives a lot of problem when somebody is trying to apply Pandas UDF for dataset where any group is more than 2^32(-1) bytes what is equal to 2gb. Solving this problem will help to use more data per Pandas UDF groupping - 2^64(-1) bytes.
Bug: Spark SQL uses the function[ length()] to return the length of the string rather than the length of the character
Status: Open | Priority: Major | Resolution: 
Created: 09/Mar/21 01:27
Affects Version/s: 3.1.1

Such as the title.

For the function length (). MySQL and other relational databases, we get the byte length, but spark SQL gets the string length. For these two cases, please provide a new function to get the byte length, otherwise it is easy to mislead users

----------------------------------------------------------------------------------------
{code:java}
// code placeholder
SparkSession.builder()
  .config(new SparkConf().setMaster("local"))
  .getOrCreate()
  .sql("select length('测a')")
  .show()

{code}
 

[result]

+-----------+
|length(测a)|

+-----------+
|2|

+-----------+

in mysql 

+-----------+
|length(测a)|

+-----------+
|4|

+-----------+
Bug: Spark app on k8s doesn't terminate without call to sparkContext.stop() method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/21 13:58
Affects Version/s: 3.1.1

Hello!
 I have run into a problem that if I don't call the method sparkContext.stop() explicitly, then a Spark driver process doesn't terminate even after its Main method has been completed. This behaviour is different from spark on yarn, where the manual sparkContext stopping is not required.
 It looks like, the problem is in using non-daemon threads, which prevent the driver jvm process from terminating.
 At least I see two non-daemon threads, if I don't call sparkContext.stop():
{code:java}
Thread[OkHttp kubernetes.default.svc,5,main]
Thread[OkHttp kubernetes.default.svc Writer,5,main]
{code}
Could you tell please, if it is possible to solve this problem?

Docker image from the official release of spark-3.1.1 hadoop3.2 is used.
Bug: Set up a workflow for developers to run benchmark in their fork
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/21 07:52
Affects Version/s: 3.2.0

Please refer the discussion in https://github.com/apache/spark/pull/31917.
Currently we don't have a pined environment to run the benchmark, and it's difficult to track and follow the performance differences.

It would be great if we have an easy way to get the benchmark results like "Running tests in your forked repository using GitHub Actions" in https://spark.apache.org/developer-tools.html
Bug: Replace the tag of release to the hash to hide RC tags in Binder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/21 01:11
Affects Version/s: 3.1.1

https://mybinder.org/v2/gh/apache/spark/v3.1.1-rc3?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart.ipynb

currently shows `spark-3.1.1-rc3`. It should be best to hide this.
Bug: unable to access grouping column after groupBy
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Mar/21 13:34
Affects Version/s: 3.1.1

When I group by nested column, I am unable to reference it after groupBy operation.

 Example:
 1) Preparing dataframe with nested column:
{code:scala}
case class Sub(a2: String)
case class Top(a1: String, s: Sub)

val s = Seq(
    Top("r1", Sub("s1")),
    Top("r2", Sub("s3"))
)
val df = s.toDF

df.printSchema
// root
//  |-- a1: string (nullable = true)
//  |-- s: struct (nullable = true)
//  |    |-- a2: string (nullable = true)
{code}
2) try to access grouping column after groupBy:
{code:scala}
df.groupBy($"s.a2").count.select('a2)
// org.apache.spark.sql.AnalysisException: cannot resolve '`a2`' given input columns: [count, s.a2];
{code}
Bug: Fix Python UDF used in GroupedAggPandasUDFTests.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/21 23:16
Affects Version/s: 2.4.7

The Python UDF {{plus_one}} used in {{GroupedAggPandasUDFTests}} is always returning {{v + 1}} regardless of its type. The return type of the UDF is 'double', so if the input is {{int}}, the result will be {{null}}.
Bug: Flaky test: TaskContextTestsWithWorkerReuse.test_task_context_correct_with_python_worker_reuse
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/21 07:12
Affects Version/s: 3.2.0

{code}

======================================================================
ERROR [1.798s]: test_task_context_correct_with_python_worker_reuse (pyspark.tests.test_taskcontext.TaskContextTestsWithWorkerReuse)
Verify the task context correct when reused python worker
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/python/pyspark/tests/test_taskcontext.py", line 289, in test_task_context_correct_with_python_worker_reuse
    self.assertTrue(pid in worker_pids)
AssertionError: False is not true

----------------------------------------------------------------------
{code}
Bug: LikeSimplification should handle NULL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/21 02:20
Affects Version/s: 3.2.0

It will throw NPE if add this test to {{RegexpExpressionsSuite}}:
{code:scala}
import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
import org.apache.spark.sql.internal.SQLConf
withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
  ConstantFolding.getClass.getName.stripSuffix("$")) {
  checkEvaluation(Literal.create("foo", StringType)
    .likeAll("%foo%", Literal.create(null, StringType)), null)
}
{code}
Bug: Non-nullable aggregates can return NULL in a correlated subquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/21 13:53
Affects Version/s: 2.4.7


Test case in scalar-subquery-select.sql:

{code:title=query}
SELECT t1a,
    (SELECT count(t2d) FROM t2 WHERE t2a = t1a) count_t2,
    (SELECT count_if(t2d > 0) FROM t2 WHERE t2a = t1a) count_if_t2,
    (SELECT approx_count_distinct(t2d) FROM t2 WHERE t2a = t1a) approx_count_distinct_t2,
    (SELECT collect_list(t2d) FROM t2 WHERE t2a = t1a) collect_list_t2,
    (SELECT collect_set(t2d) FROM t2 WHERE t2a = t1a) collect_set_t2,
    (SELECT hex(count_min_sketch(t2d, 0.5d, 0.5d, 1)) FROM t2 WHERE t2a = t1a) collect_set_t2
FROM t1;
{code}

{code:title=Result}
val1a	0	0	NULL	NULL	NULL	NULL
val1a	0	0	NULL	NULL	NULL	NULL
val1a	0	0	NULL	NULL	NULL	NULL
val1a	0	0	NULL	NULL	NULL	NULL
val1b	6	6	3	[19,119,319,19,19,19]	[19,119,319]	0000000100000000000000060000000100000004000000005D8D6AB90000000000000000000000000000000400000000000000010000000000000001
val1c	2	2	2	[219,19]	[219,19]	0000000100000000000000020000000100000004000000005D8D6AB90000000000000000000000000000000100000000000000000000000000000001
val1d	0	0	NULL	NULL	NULL	NULL
val1d	0	0	NULL	NULL	NULL	NULL
val1d	0	0	NULL	NULL	NULL	NULL
val1e	1	1	1	[19]	[19]	0000000100000000000000010000000100000004000000005D8D6AB90000000000000000000000000000000100000000000000000000000000000000
val1e	1	1	1	[19]	[19]	0000000100000000000000010000000100000004000000005D8D6AB90000000000000000000000000000000100000000000000000000000000000000
val1e	1	1	1	[19]	[19]	0000000100000000000000010000000100000004000000005D8D6AB90000000000000000000000000000000100000000000000000000000000000000
{code}
Bug: NewInstance.doGenCode should not throw malformed class name error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/21 11:07
Affects Version/s: 2.4.7

Similar to SPARK-32238 and SPARK-32999, the use of {{java.lang.Class.getSimpleName}} in {{NewInstance.doGenCode}} is problematic because Scala classes may trigger {{java.lang.InternalError: Malformed class name}}.

This happens more often when using nested classes in Scala (or declaring classes in Scala REPL which implies class nesting).

Note that on newer versions of JDK the underlying malformed class name no longer reproduces (fixed in the JDK by https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8057919), so it's less of an issue there. But on JDK8u this problem still exists so we still have to fix it.
Bug: Provide the relationship between batch ID and SQL executions (and/or Jobs) in SS UI page
Status: Open | Priority: Major | Resolution: 
Created: 01/Mar/21 09:48
Affects Version/s: 3.1.1

The current SS UI page focuses to show the trends among the batches, which is great to figure out whether the streaming query is running healthy or not, and the oddness of specific batch.

One thing still bugging you is that what you can get from here is the batch ID (number), which means you have to find out related SQL executions and Jobs manually with the batch ID. It's high likely bound to the recent runs of SQL executions/Jobs so you may not need to find it with searching on lots of pages, but the fact you need to find it by yourself manually is still annoying.

It would be nice if we can provide the relationship between batch ID and SQL executions (probably Jobs as well if the space is enough) and links to these pages, like we see job page links from SQL execution page.
Bug: Recover test reports for failed GA builds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/21 08:41
Affects Version/s: 2.4.7

[https://github.com/dawidd6/action-download-artifact/commit/621becc6d7c440318382ce6f4cb776f27dd3fef3#r48726074] there was a behaviour change in the download artifact plugin and it disabled the test reporting in failed builds.
Bug: Move checkpoint resolving logic to the rule ResolveWriteToStream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/21 15:53
Affects Version/s: 3.2.0

After SPARK-34748, we have a rule ResolveWriteToStream for the analysis logic for the resolving logic of stream write plans. Based on it, we can further move the checkpoint location resolving work in the rule.
Bug: pyspark.sql.function.lit is treating '1' the same as 1
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/Mar/21 15:46
Affects Version/s: 2.4.4

In [10]: import pyspark.sql.functions as F

In [11]: df.withColumn('x', F.lit('1')==F.lit(1)).show()
+---+---+---------+----+
|  a|  b|        c|   x|
+---+---+---------+----+
|  1|  2|    x > y|true|
|  0|  4|a > b > c|true|
+---+---+---------+----+
Bug: Some UDF calls inside transform are broken
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/Mar/21 23:48
Affects Version/s: 3.1.1

Let's say I want to create a UDF to do a simple lookup on a string:

{code:java}
import org.apache.spark.sql.{functions => f}
val M = Map("a" -> "abc", "b" -> "defg")
val BM = spark.sparkContext.broadcast(M)
val LOOKUP = f.udf((s: String) => BM.value.get(s))
{code}

Now if I have the following dataframe:

{code:java}
val df = Seq(
    Tuple1(Seq("a", "b"))
).toDF("arr")
{code}

and I want to run this UDF over each element in the array, I can do:

{code:java}
df.select(f.transform($"arr", i => LOOKUP(i)).as("arr")).show(false)
{code}

This should show:

{code:java}
+-----------+
|arr        |
+-----------+
|[abc, defg]|
+-----------+
{code}
However it actually shows:

{code:java}
+-----------+
|arr        |
+-----------+
|[def, defg]|
+-----------+
{code}

It's also broken for SQL (even without DSL). This gives the same result:

{code:java}
spark.udf.register("LOOKUP",(s: String) => BM.value.get(s))
df.selectExpr("TRANSFORM(arr, a -> LOOKUP(a)) AS arr").show(false)
{code}


Note that "def" is not even in the map I'm using.

This is a big problem because it breaks existing code/UDFs. I noticed this because the job I ported from 2.4.5 to 3.1.1 seemed to be working, but was actually producing broken data.
Bug: spark2.3 can't add column in carbondata table
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 23/Mar/21 03:21
Affects Version/s: 3.1.0

when i want to add an column in carbondata table with spark2.3, but it occur a error,here it's the detail of mistake:
{code:java}
// ALTER ADD COLUMNS does not support datasource table with type org.apache.spark.sql.CarbonSource
  at org.apache.spark.sql.execution.command.AlterTableAddColumnsCommand.verifyAlterTableAddColumn(tables.scala:242)
  at org.apache.spark.sql.execution.command.AlterTableAddColumnsCommand.run(tables.scala:194)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
  at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
  at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
  at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
  at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:190)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
{code}
so,i wish it can be supported in latest version.Thanks