av9ash/GitBugs huggingface dataset:
Bug: org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;; despite of time-based window
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/Nov/21 16:11
Affects Version/s: 3.0.1

Initially posted here: [https://stackoverflow.com/questions/70062355/org-apache-spark-sql-analysisexception-non-time-based-windows-are-not-supported]

 

'm doing the window-based sorting for the Spark Structured Streaming:
 
{{val filterWindow: WindowSpec = Window  .partitionBy("key")
  .orderBy($"time")

controlDataFrame=controlDataFrame.withColumn("Make Coffee", $"value").    
  withColumn("datetime", date_trunc("second", current_timestamp())).
  withColumn("time", current_timestamp()).
  withColumn("temp_rank", rank().over(filterWindow))
  .filter(col("temp_rank") === 1)
  .drop("temp_rank").
  withColumn("digitalTwinId", lit(digitalTwinId)).
  withWatermark("datetime", "10 seconds")}}

I'm obtaining {{time}} as {{current_timestamp()}} and in schemat I see its type as {{StructField(time,TimestampType,true)}}

Why Spark 3.0 doesn't allow me to do the window operation based on it with the following exception, as the filed is clearly time-based?
 
{{21/11/22 10:34:03 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.

org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;;Window [rank(time#163) windowspecdefinition(key#150, time#163 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS temp_rank#171], [key#150], [time#163 ASC NULLS FIRST]
+- Project [key#150, value#151, Make Coffee#154, datetime#158, time#163]}}
Bug: `with as` clause got inconsistent results
Status: Open | Priority: Major | Resolution: 
Created: 19/Nov/21 03:12
Affects Version/s: 3.1.2

In Spark3.1, the `with as` clause in the same SQL is executed multiple times,  got different results

`

with tab as (
 select 'Withas' as name, rand() as rand_number
)
select name, rand_number
from tab
union all
select name, rand_number
from tab

`

!spark3.1.png!

But In spark2.3, it got consistent results

`

with tab as (
 select 'Withas' as name, rand() as rand_number
)
select name, rand_number
from tab
union all
select name, rand_number
from tab

`

!spark2.3.png!

Why does Spark3.1.2 return different results?

Has anyone encountered this problem?
Bug: WidthBucket throws NullPointerException in WholeStageCodegenExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/21 11:59
Affects Version/s: 3.2.0

Repro: Disable ConstantFolding and run
{code:java}
SELECT width_bucket(3.5, 3.0, 3.0, 888) {code}
Bug: pyspark.{ml, mllib}.linalg.Vector API
Status: Open | Priority: Major | Resolution: 
Created: 21/Nov/21 13:50
Affects Version/s: 3.3.0

At the moment both {{Vector}} implementations have minimal API. {{mllib}}

{code:python}
class Vector(object):
    __UDT__ = ....
    def toArray(self): ...
    def asML(self): ...
{code}

{{ml}}

{code:python}
class Vector(object):
    __UDT__ = ...
    def toArray(self): ...

{code}

which doesn't cover actual API being used.

This causes typing issues, when we expect any {{Vector}}.
Bug: Add tree pattern pruning to CTESubstitution rule
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/21 00:04
Affects Version/s: 3.1.0

I propose to add tree pattern pruning to the CTESubstitution rule in order to skip tree traversal when the tree does not contain an UnresolvedWith expression.

This is motivated by profiling a job which uses DataFrame APIs to incrementally construct a huge query plan (200k+ nodes): each API call results in eager re-analysis of the plan, of which CTESubstitution accounts for the majority of the analysis time. This query didn't contain CTEs, so skipping the CTESubstitution significantly speeds up analysis.

I plan to submit a patch for this.
Bug: Improve SQL Rest API Parsing
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Nov/21 01:10
Affects Version/s: 3.2.0

https://issues.apache.org/jira/browse/SPARK-31440 added improvements for SQL Rest API. Currently, values like
{code:java}
"value" : "total (min, med, max (stageId: taskId))\n177.0 B (59.0 B, 59.0 B, 59.0 B (stage 1.0: task 5))"{code}
 are currently shown from Rest API calls which are not easily digested in its current form.New processing logic of the values is introduced to organize and process new metric fields in a more user friendly manner.
Bug: Spark K8s tests are broken
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Nov/21 20:47
Affects Version/s: 3.2.0

For decom: We should look for "Decommission executors" instead of "Remove reason statistics:", the listenerbus doesn't have strong guarantees and depending on it makes this test flaky.

 

For R : It has not passed in weeks, lets disable it.
Bug: Spark Thrift Server (STS) driver fullFC becourse of timeoutExecutor not shutdown correctly
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Nov/21 11:32
Affects Version/s: 3.1.0

When spark.sql.thriftServer.queryTimeout or java.sql.Statement.setQueryTimeout is setted >0 , SparkExecuteStatementOperation add timeoutExecutor to kill time-consumed query in [SPARK-26533|https://issues.apache.org/jira/browse/SPARK-26533]. But timeoutExecutor is not shutdown correctly when statement is finished, it can only be shutdown when timeout. When we set timeout to a long time for example 1 hour, the long-running STS driver will FullGC and the application is not available for  a long time.
Bug: Add java.security.jgss/sun.security.krb5 to DEFAULT_MODULE_OPTIONS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/21 10:06
Affects Version/s: 3.3.0


{noformat}
Exception in thread "main" java.lang.IllegalArgumentException: Can't get Kerberos realm
	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
	at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:50)
	at org.apache.spark.deploy.SparkHadoopUtil$.instance$lzycompute(SparkHadoopUtil.scala:397)
	at org.apache.spark.deploy.SparkHadoopUtil$.instance(SparkHadoopUtil.scala:397)
	at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:418)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:423)
	at org.apache.spark.executor.YarnCoarseGrainedExecutorBackend$.main(YarnCoarseGrainedExecutorBackend.scala:81)
	at org.apache.spark.executor.YarnCoarseGrainedExecutorBackend.main(YarnCoarseGrainedExecutorBackend.scala)
Caused by: java.lang.IllegalAccessException: class org.apache.hadoop.security.authentication.util.KerberosUtil cannot access class sun.security.krb5.Config (in module java.security.jgss) because module java.security.jgss does not export sun.security.krb5 to unnamed module @3a0baae5
	at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392)
	at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674)
	at java.base/java.lang.reflect.Method.invoke(Method.java:560)
	at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:85)
	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
	... 9 more
{noformat}
Bug: File system delegation tokens are leaked
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Nov/21 07:45
Affects Version/s: 2.4.0

On a very busy Hadoop cluster (with HDFS at rest encryption) we found KMS accumulated millions of delegation tokens that are not cancelled even after jobs are finished, and KMS goes out of memory within a day because of the delegation token leak.

We were able to reproduce the bug in a smaller test cluster, and realized when a Spark job starts, it acquires two delegation tokens, and only one is cancelled properly after the job finishes. The other one is left over and linger around for up to 7 days ( default Hadoop delegation token life time).

YARN handles the lifecycle of a delegation token properly if its renewer is 'yarn'. However, Spark intentionally (a hack?) acquires a second delegation token with the job issuer as the renewer, simply to get the token renewal interval. The token is then ignored but not cancelled.

Propose: cancel the delegation token immediately after the token renewal interval is obtained.

Environment: CDH6.3.2 (based on Apache Spark 2.4.0) but the bug probably got introduced since day 1.
Bug: Display StageIds in Operators for SQL UI
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Nov/21 22:45
Affects Version/s: 3.2.0

This proposes a more generalized solution of https://issues.apache.org/jira/browse/SPARK-30209, where a stageId-> operator mapping is done with the following algorithm.

 1. Read SparkGraph to get every Node's name and respective AccumulatorIDs.
 2. Gets each stage's AccumulatorIDs.
 3. Maps Operators to stages by checking for non-zero intersection of Step 1 and 2's AccumulatorIDs.
 4. Connect SparkGraphNodes to respective StageIDs for rendering in SQL UI.

As a result, some operators without max metrics values will also have stageIds in the UI. This Jira also aims to add minor enhancements to the SQL UI tab.
Bug: split function behave differently between spark 2.3 and spark 3.2
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Nov/21 02:00
Affects Version/s: 3.1.1

while use split function in sql, it behave differently between 2.3 and 3.2, which cause incorrect problem.

we can use this sql to reproduce this problem:

 

create table split_test ( id int,name string)

insert into split_test values(1,"abc;def")

explain extended select split(name,'\\\\;') from split_test

 

spark3:

spark-sql> Explain extended select split(name,'\\\\;') from split_test;

== Parsed Logical Plan ==

'Project [unresolvedalias('split('name, \\;), None)]

+- 'UnresolvedRelation [split_test], [], false

 

spark2:

 

spark-sql> Explain extended select split(name,'\\\\;') from split_test;

== Parsed Logical Plan ==

'Project [unresolvedalias('split('name, \;), None)]

+- 'UnresolvedRelation split_test

 

It looks like the deal of escape is different
Bug: Rename (Spark)DataFrame.to_pandas_on_spark to (Spark)DataFrame.pandas_api
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 15/Nov/21 19:47
Affects Version/s: 3.3.0

Currently, (Spark)DataFrame.to_pandas_on_spark is too long to memorize and inconvenient to call.

So we wanted to rename to_pandas_on_spark to pandas_api for API usability
Bug: View text can only be SELECT queries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/21 08:24
Affects Version/s: 3.3.0

The current implementation of persistent view is create hive table with view text.
The view text is just a query string, so the hackers may tamper with it through various means.
Such as:
{code:java}
select * from tab1
{code}
 tampered with
 
{code:java}
drop table tab1
{code}
Bug: Fix Python linter failure in branch-3.1
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Nov/21 06:20
Affects Version/s: 3.1.0

The Python linter is failing for some reason in branch-3.1.

[https://github.com/apache/spark/runs/4151205472?check_suite_focus=true]

We should fix it
Bug: Mitigate ConcurrentModificationException thrown from a test in MLEventSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/21 15:36
Affects Version/s: 3.3.0

Recently, I notice ConcurrentModificationException is sometimes thrown from the following part of the test "pipeline read/write events" in MLEventSuite when Scala 2.13 is used.
{code}
events.map(JsonProtocol.sparkEventToJson).foreach { event =>
  assert(JsonProtocol.sparkEventFromJson(event).isInstanceOf[MLEvent])
}
{code}

I think the root cause is the ArrayBuffer (events) is updated asynchronously by the following part.
{code}
private val listener: SparkListener = new SparkListener {
  override def onOtherEvent(event: SparkListenerEvent): Unit = event match {
    case e: MLEvent => events.append(e)
    case _ =>
  }
}
{code}
Bug: Upgrade kubernetes-client to 5.10.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/21 04:13
Affects Version/s: 3.3.0

kubernetes-client 5.10.0 and 5.10.1 were released, which include some bug fixes.

https://github.com/fabric8io/kubernetes-client/releases/tag/v5.10.0
https://github.com/fabric8io/kubernetes-client/releases/tag/v5.10.1

Especially, the connection leak issue would affect Spark.
https://github.com/fabric8io/kubernetes-client/issues/3561
Bug: Unify v1 and v2 ALTER TABLE .. REPLACE COLUMNS tests
Status: Open | Priority: Major | Resolution: 
Created: 12/Nov/21 14:17
Affects Version/s: 3.3.0

Extract ALTER TABLE .. REPLACE COLUMNS tests to the common place to run them for V1 and v2 datasources. Some tests can be places to V1 and V2 specific test suites.
Bug: Spark Dynamic Resource Allocation in Standalone Mode
Status: Open | Priority: Major | Resolution: 
Created: 12/Nov/21 11:37
Affects Version/s: 3.1.1

Hello, 

I have some questions about whether it is possible to use "Dynamic Resource Allocation" in standalone mode (2 workers and 1 master). I have been trying the suggestions proposed in your code, either by modifying by adding configuration variables to the SparkSession, or by including them directly as environment variables in Docker. None of these modifications have worked, and until one task is finished, I don't have the resources to do the next one (FIFO mode tasks). I wanted to ask if there is a way to run the tasks in parallel between several workers in standalone mode, since according to your website, it is possible, but it is disabled by default. Following the official documentation, I understand that it is possible, but realising the changes, I don't see any change in the concurrency: 

_Standalone mode: By default, applications submitted to the standalone mode cluster will run in FIFO (first-in-first-out) order, and each application will try to use all available nodes. You can limit the number of nodes an application uses by setting the spark.cores.max configuration property in it, or change the default for applications that don’t set this setting through spark.deploy.defaultCores. Finally, in addition to controlling cores, each application’s spark.executor.memory setting controls its memory use._
Bug: Check inserting of ANSI intervals into a table partitioned by the interval columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 19:03
Affects Version/s: 3.3.0

Write tests that check INSERT INTO (statically and dynamically) of ANSI intervals to a table that is partitioned by the intervals. Need to check V1/V2 In-Memory catalogs.
Bug: Remove explicit GC options from Scala tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 16:49
Affects Version/s: 3.3.0

At Spark 3.0, SPARK-29282 introduced the explicit GC options in Scala tests to see the consistent result between Java8/Java11.

Now, we have Java 17 additionally. This PR aims to use the default GC options for each JVM.

This is a partial revert of SPARK-29282.
Bug: OptimizeSkewInRebalancePartitions support optimize non-root node
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/21 08:41
Affects Version/s: 3.3.0

`OptimizeSkewInRebalancePartitions` now is applied if the `RebalancePartitions` is the root node, but sometimes, we expect a local sort after do RebalancePartitions that can improve the compression ratio.

After SPARK-36184, we make validate easy that whether the rule introduces extra shuffle or not and the output partitioning is ensured by `AQEShuffleReadExec.outputPartitioning`.

So it is safe to make `OptimizeSkewInRebalancePartitions` support optimize non-root node.
Bug: Spark SQL reads unnecessary nested fields (filter after explode)
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Nov/21 01:40
Affects Version/s: 3.2.0

In this example, reading unnecessary nested fields still happens.

Data preparation:
{code:java}
case class Struct(v1: String, v2: String, v3: String)
case class Event(struct: Struct, array: Seq[String])

Seq(
  Event(Struct("v1","v2","v3"), Seq("cx1", "cx2"))
).toDF().write.mode("overwrite").saveAsTable("table")
{code}
 v2 and v3 columns aren't needed here, but still exist in the physical plan.
{code:java}
spark.table("table")
  .select($"struct.v1", explode($"array").as("el"))
  .filter($"el" === "cx1")
  .explain(true)
 
== Physical Plan ==
... ReadSchema: struct<struct:struct<v1:string,v2:string,v3:string>,array:array<string>>

{code}
If you just remove _filter_ or move _explode_ to second _select_, everything is fine:
{code:java}
spark.table("table")
  .select($"struct.v1", explode($"array").as("el"))
  //.filter($"el" === "cx1")
  .explain(true)
  
// ... ReadSchema: struct<struct:struct<v1:string>,array:array<string>>

spark.table("table")
  .select($"struct.v1", $"array")
  .select($"v1", explode($"array").as("el"))
  .filter($"el" === "cx1")
  .explain(true)
  
// ... ReadSchema: struct<struct:struct<v1:string>,array:array<string>>
{code}
 

*Yet another example: left_anti join after double select:*
{code:java}
case class Struct(v1: String, v2: String, v3: String)
case class Event(struct: Struct, field1: String, field2: String)
Seq(
      Event(Struct("v1","v2","v3"), "fld1", "fld2")
    ).toDF().write.mode("overwrite").saveAsTable("table")    
val joinDf = Seq("id1").toDF("id")

spark.table("table")
      .select("struct", "field1")
      .select($"struct.v1", $"field1")
      .join(joinDf, $"field1" === joinDf("id"), "left_anti")
      .explain(true)

// ===> ReadSchema: struct<struct:struct<v1:string,v2:string,v3:string>,field1:string>
{code}
Instead of the first select, it can be other types of manipulations with the original df, for example {color:#00875a}.withColumn("field3", lit("f3")){color} or {color:#00875a}.drop("field2"){color}, which will also lead to reading unnecessary nested fields from _struct_.

But if you just remove the first select or change type of join, reading nested fields will be correct:
{code:java}
// .select("struct", "field1")
===> ReadSchema: struct<struct:struct<v1:string>,field1:string>

.join(joinDf, $"field1" === joinDf("id"), "left")
===> ReadSchema: struct<struct:struct<v1:string>,field1:string>
{code}
PS: The first select might look strange in the context of this example, but in a real system, it might be part of a common api, that other parts of the system use with their own expressions on top of this api.
Bug: The collect-like API in SparkPlan should support columnar output
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Nov/21 01:45
Affects Version/s: 3.3.0

Currently, the collect-like APIs in SparkPlan, e.g. executeCollect, do not work if the plan is columnar execution.

We should extend the API coverage to all execution modes.
Bug: Run PyArrow tests on Python 3.10
Status: Reopened | Priority: Major | Resolution: 
Created: 08/Nov/21 23:54
Affects Version/s: 3.3.0

Though pyarrow doesn’t support python 3.10 yet [https://pypi.org/project/pyarrow/,] we wanted to adjust PySpark codes gradually to support Python 3.10.
Bug: Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests)
Status: Closed | Priority: Major | Resolution: Invalid
Created: 09/Nov/21 00:06
Affects Version/s: 3.3.0

{code:java}
  File "/Users/xinrong.meng/spark/python/pyspark/sql/tests/test_dataframe.py", line 957, in test_create_nan_decimal_dataframe
    self.spark.createDataFrame(data=[Decimal('NaN')], schema='decimal').collect(),
  File "/Users/xinrong.meng/spark/python/pyspark/sql/dataframe.py", line 751, in collect
    sock_info = self._jdf.collectToPython()
  File "/Users/xinrong.meng/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1309, in __call__
    return_value = get_return_value(
  File "/Users/xinrong.meng/spark/python/pyspark/sql/utils.py", line 178, in deco
    return f(*a, **kw)
  File "/Users/xinrong.meng/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o135.collectToPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 2.0 failed 1 times, most recent failure: Lost task 3.0 in stage 2.0 (TID 7) (172.16.203.223 executor driver): net.razorvine.pickle.PickleException: problem construction object: java.lang.reflect.InvocationTargetException
...{code}
Bug: Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests)
Status: Closed | Priority: Major | Resolution: Invalid
Created: 09/Nov/21 00:10
Affects Version/s: 3.3.0

{code:java}
Traceback (most recent call last):
  File "/Users/xinrong.meng/spark/python/pyspark/sql/tests/test_functions.py", line 247, in test_make_date
    row_from_col = df.select(make_date(df.Y, df.M, df.D)).first()
  File "/Users/xinrong.meng/spark/python/pyspark/sql/functions.py", line 2159, in make_date
    jc = sc._jvm.functions.make_date(year_col, month_col, day_col)
  File "/Users/xinrong.meng/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1535, in __getattr__
    raise Py4JError(
py4j.protocol.Py4JError: org.apache.spark.sql.functions.make_date does not exist in the JVM
 {code}
Bug: Failed test_capture_user_friendly_exception (pyspark.sql.tests.test_utils.UtilsTests)
Status: Closed | Priority: Major | Resolution: Invalid
Created: 09/Nov/21 00:17
Affects Version/s: 3.3.0

 
{code:java}
Traceback (most recent call last):
  File "/Users/xinrong.meng/spark/python/pyspark/sql/tests/test_utils.py", line 34, in test_capture_user_friendly_exception
    self.spark.sql("select `中文字段`")
pyspark.sql.utils.AnalysisException: cannot resolve '`中文字段`' given input columns: []; line 1 7;
'Project ['中文字段]
+- OneRowRelation {code}
Bug: Failed _joinAsOf doctest
Status: Closed | Priority: Major | Resolution: Invalid
Created: 09/Nov/21 00:23
Affects Version/s: 3.3.0

{code:java}
File "/Users/xinrong.meng/spark/python/pyspark/sql/dataframe.py", line 1523, in pyspark.sql.dataframe.DataFrame._joinAsOf
Failed example:
    left._joinAsOf(
        right, leftAsOfColumn="a", rightAsOfColumn="a", how="left", tolerance=F.lit(1)
    ).select(left.a, 'left_val', 'right_val').sort("a").collect()
Exception raised:
    Traceback (most recent call last):
      File "/opt/miniconda3/envs/py10/lib/python3.10/doctest.py", line 1348, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.dataframe.DataFrame._joinAsOf[5]>", line 1, in <module>
        left._joinAsOf(
      File "/Users/xinrong.meng/spark/python/pyspark/sql/dataframe.py", line 1578, in _joinAsOf
        jdf = self._jdf.joinAsOf(
      File "/Users/xinrong.meng/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1309, in __call__
        return_value = get_return_value(
      File "/Users/xinrong.meng/spark/python/pyspark/sql/utils.py", line 178, in deco
        return f(*a, **kw)
      File "/Users/xinrong.meng/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 330, in get_return_value
        raise Py4JError(
    py4j.protocol.Py4JError: An error occurred while calling o283.joinAsOf. Trace:
    py4j.Py4JException: Method joinAsOf([class org.apache.spark.sql.Dataset, class org.apache.spark.sql.Column, class org.apache.spark.sql.Column, null, class java.lang.String, class org.apache.spark.sql.Column, class java.lang.Boolean, class java.lang.String]) does not exist
 {code}