av9ash/GitBugs huggingface dataset:
Bug: Avoid printing full Exception stack trace, if HBase/Kafka/Hive services are not running in a secure cluster 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jun/21 03:37
Affects Version/s: 3.0.0

In a secure Yarn cluster, even though HBase or Kafka, or Hive services are not used in the user application, yarn client unnecessarily trying to generate  Delegations token from these services. This will add additional delays while submitting spark application in a yarn cluster

 Also during HBase delegation, token generation step in the application submit stage,  HBaseDelegationTokenProvider prints a full Exception Stack trace and it causes a noisy warning. 
{code:java}
WARN security.HBaseDelegationTokenProvider: Failed to get token from service hbase
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn(HBaseDelegationT
okenProvider.scala:93)

more than 100+ line exception stack trace{code}
Hence, if these services are not used in the user Application, it is better add WARN message to disable Delegation Token generation for those services. 

ie, spark.security.credentials.hbase.enabled=false , 

 spark.security.credentials.hive.enabled=false ,

 spark.security.credentials.kafka.enabled=false
Bug: CatalogFileIndex only used if Partition Columns Nonempty
Status: Open | Priority: Minor | Resolution: 
Created: 23/Jun/21 16:48
Affects Version/s: 3.1.2

Currently, when deciding whether to use a CatalogFileIndex, we gate on catalogTable.get.partitionColumnNames.nonEmpty ([see here|[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L398])] I believe what we actually want is to check that the catalogTable.get.dataSchema.nonEmpty, as I don't think it's actually necessary that a table have partition columns in order to be read from a CatalogFileIndex. This isn't a correctness issue, just a missed optimization any time there are no partition columns for a table.

 

Palantir [fixed this in our fork|https://github.com/palantir/spark/commit/e040ff5bf4d1b2d37264ad19468e0892c63b9798] a long time ago, but I don't think anyone ever remembered to push it upstream.
Bug: Move new interval type test cases from CastSuite to CastBaseSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jun/21 12:08
Affects Version/s: 3.2.0

There are a few test cases that are supposed to be in CastSuiteBase instead CastSuite:
* SPARK-35112: Cast string to day-time interval
* SPARK-35111: Cast string to year-month interval
* SPARK-35820: Support cast DayTimeIntervalType in different fields
* SPARK-35819: Support cast YearMonthIntervalType in different fields
Bug: Ensure all modules can be maven test independently in Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jun/21 06:48
Affects Version/s: 3.2.0

 

Execute

 
{code:java}
mvn clean install -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl external/kafka-0-10-sql
{code}
 

1 scala test aborted, the error message is 
{code:java}
Discovery starting.
Discovery completed in 857 milliseconds.
Run starting. Expected test count is: 464
...
KafkaRelationSuiteV2:
- explicit earliest to latest offsets
- default starting and ending offsets
- explicit offsets
- default starting and ending offsets with headers
- timestamp provided for starting and ending
- timestamp provided for starting, offset provided for ending
- timestamp provided for ending, offset provided for starting
- timestamp provided for starting, ending not provided
- timestamp provided for ending, starting not provided
- global timestamp provided for starting and ending
- no matched offset for timestamp - startingOffsets
- preferences on offset related options
- no matched offset for timestamp - endingOffsets
*** RUN ABORTED ***
  java.lang.NoClassDefFoundError: scala/collection/parallel/TaskSupport
  at org.apache.spark.SparkContext.$anonfun$union$1(SparkContext.scala:1411)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.SparkContext.withScope(SparkContext.scala:788)
  at org.apache.spark.SparkContext.union(SparkContext.scala:1405)
  at org.apache.spark.sql.execution.UnionExec.doExecute(basicPhysicalOperators.scala:697)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:182)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:220)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:217)
  ...
  Cause: java.lang.ClassNotFoundException: scala.collection.parallel.TaskSupport
  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
  at org.apache.spark.SparkContext.$anonfun$union$1(SparkContext.scala:1411)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.SparkContext.withScope(SparkContext.scala:788)
  at org.apache.spark.SparkContext.union(SparkContext.scala:1405)
  at org.apache.spark.sql.execution.UnionExec.doExecute(basicPhysicalOperators.scala:697)
  ...

{code}
Bug: Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jun/21 17:23
Affects Version/s: 3.1.2

Metric {{allUpdatesTimeMs}} meant to capture the start to end walltime of the operator {{FlatMapGroupsWithStateExec}}, but currently it just [captures|https://github.com/apache/spark/blob/79362c4efcb6bd4b575438330a14a6191cca5e4b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala#L121] the iterator creation time. 

Fix it to measure similar to how other stateful operators measure. Example one [here|https://github.com/apache/spark/blob/79362c4efcb6bd4b575438330a14a6191cca5e4b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L406]. This measurement is not perfect due to the nature of the lazy iterator and also includes the time the consumer operator spent in processing the current operator output, but it should give a good signal when comparing the metric in one microbatch to the metric in another microbatch.
Bug: Improve error message when creating k8s pod failed.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jun/21 13:31
Affects Version/s: 3.1.2

 

If I use wrong k8s master URL, I will get the below error message.

I think we could improve error message, so end-users can get a better understanding of the message.
{code:java}
(base) ➜ spark git:(master) ./bin/spark-submit \
--master k8s://https://192.168.49.3:8443 \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=3 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.container.image=pingsutw/spark:testing \
local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0-SNAPSHOT.jar
21/06/09 20:50:37 WARN Utils: Your hostname, kobe-pc resolves to a loopback address: 127.0.1.1; using 192.168.103.20 instead (on interface ens160)
21/06/09 20:50:37 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
21/06/09 20:50:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/06/09 20:50:38 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
21/06/09 20:50:39 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [default] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:380) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:380)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:86) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:86)
{code}
Bug: Ignore all ".idea" directories 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jun/21 09:32
Affects Version/s: 3.2.0

After https://github.com/apache/spark/pull/32337, all the `.idea/` in submodules are treated as git difference again.
For example, when I open the project `resource-managers/yarn/` with IntelliJ, the git status becomes

{code:java}
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	resource-managers/yarn/.idea/
{code}
The same issue happens on opening `sql/hive-thriftserver/` with IntelliJ.
We should ignore all the ".idea" directories instead of the one under the root path.
Bug: UT `handles k8s cluster mode` fails on MacOs >= 10.15
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jun/21 10:15
Affects Version/s: 3.0.3

When I run SparkSubmitSuite on MacOs 10.15.7, I got AssertionError for `handles k8s cluster mode` test after pr [SPARK-35691|https://issues.apache.org/jira/browse/SPARK-35691] due to `File(path).getCanonicalFile().toURI()` function  with absolute path as parameter will return path begin with /System/Volumes/Data.

eg.  /home/testjars.jar will get 

[file:/System/Volumes/Data/home/testjars.jar|file:///System/Volumes/Data/home/testjars.jar]
Bug: Add BaseTest for DataTypeOps
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jun/21 08:22
Affects Version/s: 3.2.0

When complete https://github.com/apache/spark/pull/32821, I found there are no test for DataTypeOps. There were many logic when DataTypeOps loaded, it's better to add the test to make sure interface stable.
Bug: CoalesceExec can execute child plan twice
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jun/21 16:13
Affects Version/s: 2.3.0

CoalesceExec calls `child.execute()` in the if condition and throws away the results, then calls `child.execute()` again in the else condition. This could cause a section of the plan to be executed twice.
{code:java}
protected override def doExecute(): RDD[InternalRow] = {
  if (numPartitions == 1 && child.execute().getNumPartitions < 1) {
    // Make sure we don't output an RDD with 0 partitions, when claiming that we have a
    // `SinglePartition`.
    new CoalesceExec.EmptyRDDWithPartitions(sparkContext, numPartitions)
  } else {
    child.execute().coalesce(numPartitions, shuffle = false)
  }
} {code}
Bug: Release on-going map properly for NULL-aware ANTI join
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jun/21 23:58
Affects Version/s: 3.2.0

NULL-aware ANTI join (https://issues.apache.org/jira/browse/SPARK-32290) detects NULL join keys during building the map for `HashedRelation`, and will immediately return `HashedRelationWithAllNullKeys` without taking care of the map built already. Before returning `HashedRelationWithAllNullKeys`, the map needs to be freed properly to save memory and keep memory accounting correctly.
Bug: addFile/addJar/addDirectory should put CanonicalFile
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jun/21 06:24
Affects Version/s: 3.2.0

I met the error below.
  
{code:java}
 21/06/07 00:06:57 ERROR SparkContext: Failed to add file:/home/runner/work/spark/spark/./core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar to Spark environment    java.lang.IllegalArgumentException: requirement failed: File spark-core_2.12-3.2.0-SNAPSHOT-tests.jar was already registered with a different path (old path = /home/runner/work/spark/spark/core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar, new path = /home/runner/work/spark/spark/./core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar {code}
But actually, *[{color:#172b4d}/home/runner/work/spark/spark/./core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar*{color}|file:///home/runner/work/spark/spark/core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar*]* and ** /*home/runner/work/spark/spark/core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar are the same**.

I think we should put the Canonical File in ConcurrentHashMap.

[https://github.com/apache/spark/blob/9f010a8eb20502292b3bca42d17ce1dc357343b1/core/src/main/scala/org/apache/spark/rpc/netty/NettyStreamManager.scala#L68-L89]
Bug: Fix the max rows check for broadcast exchange
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jun/21 21:45
Affects Version/s: 3.2.0

This is to fix the maximal allowed number of rows check in `BroadcastExchangeExec`. After [https://github.com/apache/spark/pull/27828,] the max number of rows is calculated based on max capacity of `BytesToBytesMap` (previous value before the PR is 512000000). This calculation is not accurate as only `UnsafeHashedRelation` is using `BytesToBytesMap`. `LongHashedRelation` (used for broadcast join on key with long data type) has limit of 512000000 ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L584] ), and `BroadcastNestedLoopJoinExec` is not depending on `HashedRelation` at all.
Bug: Break down CastSuite/AnsiCastSuite into multiple files
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jun/21 14:27
Affects Version/s: 3.2.0

Currently, the file CastSuite.scala becomes big, 2000 lines, 2 base classes, 4 test suites.
In my previous work of Timestamp without time zone, I planned to put new test cases in CastSuiteBase, but they were accidentally added in AnsiCastSuiteBase.

To make development and review easier, I would like to break the file down into files.
Bug: wait until something does get queued
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 11/Jun/21 05:09
Affects Version/s: 3.0.0

 If nothing has been added to the queue, should wait until something does get queued instead of loop after timeout, It prevents an ineffective loop.

 

Currently, We will cotinue loop after timeout of the 100ms if the queue is empty,  It prevents an ineffective cycle.

eg: 

1. referenceQueue.remove(timeout), if the queue is empty

2. remove will return directly after wait timeout

3. continue the loop.

reference
 # Spark ContextCleaner

{code:java}
while (!stopped) {
  try {
    val reference = Option(referenceQueue.remove(ContextCleaner.REF_QUEUE_POLL_TIMEOUT))
      .map(_.asInstanceOf[CleanupTaskWeakReference])
...
}
...
}{code}
     2. JDK ReferenceQueue
{code:java}
public Reference<? extends T> remove(long timeout)
    throws IllegalArgumentException, InterruptedException
{
    if (timeout < 0) {
        throw new IllegalArgumentException("Negative timeout value");
    }
    synchronized (lock) {
        Reference<? extends T> r = reallyPoll();
        if (r != null) return r;
        long start = (timeout == 0) ? 0 : System.nanoTime();
        for (;;) {
            lock.wait(timeout); 
            r = reallyPoll();
            if (r != null) return r;
            if (timeout != 0) {
                long end = System.nanoTime();
                timeout -= (end - start) / 1000_000;
                if (timeout <= 0) return null;
                start = end;
            }
        }
    }
}
{code}
 If we have not timeout, Thread will wait until Reference.enqueue be called. i.e. something that need to be cleanUp be add to the queue.
Bug: Support Joint eviction strategies for cached RDD partitions
Status: In Progress | Priority: Minor | Resolution: 
Created: 14/Jun/21 06:01
Affects Version/s: 3.1.2

As of now, there's as it were ousting procedure for cached RDD segment in Spark. The default RDD removal methodology is LRU .When memory space not adequate for RDD caching, a few allotments will be ousted, on the off chance that these segments are utilized once more latterly, they will be replicated by the Ancestry data and cached in memory once more. The replicate stage will bring in extra taken a toll. Be that as it may, LRU has no ensure for the most reduced duplicate cost. The to begin with RDD that required to be cached is ordinarily created by perusing from HDFS and doing a few changes. The perusing operation ordinarily fetched longer time than other Start transformations.
Bug: Consider making the ':' in STRUCT data type definition optional
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jun/21 05:27
Affects Version/s: 3.1.2

The STRUCT type syntax is defined like this:

STRUCT(fieldNmae: fileType [NOT NULL][COMMENT stringLiteral][,.....])

So the field list is NEARLY the same as a column list

if we could make ':' optional it would be so much cleaner an less proprietary
Bug: Stream-stream join keys should be reordered properly
Status: In Progress | Priority: Minor | Resolution: 
Created: 09/Jun/21 05:33
Affects Version/s: 3.2.0

For query with multiple stream-stream joins, unnecessary shuffle can be happened as we don't reorder join keys for stream-stream join. We already reorder join keys for shuffled hash join and sort merge join, so we can also have it for stream-stream join - [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala#L232] .
Bug: Spark creates only _SUCCESS file after empty dataFrame is saved as parquet for partitioned data
Status: In Progress | Priority: Minor | Resolution: 
Created: 01/Jun/21 10:44
Affects Version/s: 2.4.0

Whenever an empty dataframe is saved as a parquet file with partitions, the target directory only contains _SUCCESS file.

Assuming, the dataframe has 3 columns:
 some_column_1, some_column_2, some_partition_column_1

and the target location for dataframe is /user/spark/df_name

*Current Result*:  /user/spark/df_name/_SUCCESS

*Expected Result*: /user/spark/df_name/some_partition_column_1=_HIVE_DEFAULT_PARTITION_/<some_spark_generated_file_name>.snappy.parquet

where that parquet file will have the schema for the data.

This approach makes sure that any job reading this data doesn't get failed due to:
 Exception: org.apache.spark.sql.AnalysisException: Unable to infer schema for Parquet. It must be specified manually.

*Steps for reproduce (Scala)*:

 
{code:java}
// create an empty DF with schema
val inputDF = Seq(
  ("value1", "value2", "partition1"),
  ("value3", "value4", "partition2"))
  .toDF("some_column_1", "some_column_2", "some_partition_column_1")
  .where("1==2")

// write dataframe into partitions
inputDF.write
  .partitionBy("some_partition_column_1")
  .mode(SaveMode.Overwrite)
  .parquet("/user/spark/df_name")


// Read dataframe
// Exception: org.apache.spark.sql.AnalysisException: Unable to infer schema for // Parquet. It must be specified manually.
val readDF = spark.read.parquet("/user/spark/df_name")
{code}
 

Similar issue was created under below Jira ticket but it was only for non-partitioned data.

https://issues.apache.org/jira/browse/SPARK-23271

We need a similar implementation for partitioned target as well.
Bug: SELECT rand(cast(3 / 7 AS int)) value gives different resultset in Spark 3.1 as compared to Spark 2.4.5
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 03/Jun/21 14:54
Affects Version/s: 3.1.1

Issue - SELECT rand(cast(3 / 7 AS int)) value gives different resultset in Spark 3.1 as compared to Spark 2.4.5

Queries are executed in Spark beeline.

 

Output in Spark 3.1 
0: jdbc:hive2://10.20.253.239:23040/default> SELECT rand(cast(3 / 7 AS int));
+----------------------------------------------------+
| rand(CAST((CAST(3 AS DOUBLE) / CAST(7 AS DOUBLE)) AS INT)) |
+----------------------------------------------------+
| 0.7604953758285915 |
+----------------------------------------------------+
1 row selected (0.042 seconds)


Output in  Spark 2.4.5
0: jdbc:hive2://linux-63:22550/> SELECT rand(cast(3 / 7 AS int));
+-------------------------------------------------------------+
| rand(CAST((CAST(3 AS DOUBLE) / CAST(7 AS DOUBLE)) AS INT)) |
+-------------------------------------------------------------+
| 0.8446490682263027 |
+-------------------------------------------------------------+
1 row selected (6.358 seconds)
Bug: 2 active tasks shown in Spark UI but executor core is 1 and task cpu is 1
Status: Open | Priority: Minor | Resolution: 
Created: 03/Jun/21 20:13
Affects Version/s: 3.0.2

We see an incorrect element on UI. We set executor core as 1, and task cpu as 1, but we sometimes see 2 active tasks for one executor in "Executors" tab on Spark UI.

It seems to be UI-only issue, because from the event timeline diagram, two tasks are not actually run at the same time. The second one begins to run after first one finishes.

Note that two tasks have very short duration 0.7s and 1s.

So it looks like there is a delay on UI to reflect correct active tasks.
Bug: Upgrade SBT to 1.5.3
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Jun/21 19:33
Affects Version/s: 3.2.0

Recently, SBT 1.5.3 was released and includes a bug fix for Scala 2.13.16+ and Scala 3.
https://github.com/sbt/sbt/releases/tag/v1.5.3
Bug: Use better exception type if database doesn't exist on `drop database`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jun/21 11:10
Affects Version/s: 3.2.0

Curently execute `drop database test` will throw unfriendly error msg.

{code:java}
Error in query: org.apache.hadoop.hive.metastore.api.NoSuchObjectException: test
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException: test
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:112)
	at org.apache.spark.sql.hive.HiveExternalCatalog.dropDatabase(HiveExternalCatalog.scala:200)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.dropDatabase(ExternalCatalogWithListener.scala:53)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.dropDatabase(SessionCatalog.scala:273)
	at org.apache.spark.sql.execution.command.DropDatabaseCommand.run(ddl.scala:111)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
	at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3707)
{code}