av9ash/GitBugs huggingface dataset:
Bug: metadata mismatch for StructField when running some tests.
Status: Reopened | Priority: Major | Resolution: 
Created: 29/Mar/23 15:31
Affects Version/s: 3.5.0

For some reason, the metadata of `StructField` is different in a few tests when using Spark Connect. However, the function works properly.

For example, when running `python/run-tests --testnames 'pyspark.pandas.tests.connect.data_type_ops.test_parity_binary_ops BinaryOpsParityTests.test_add'` it complains `AssertionError: ([InternalField(dtype=int64, struct_field=StructField('bool', LongType(), False))], [StructField('bool', LongType(), False)])` because metadata is different something like `\{'__autoGeneratedAlias': 'true'}` but they have same name, type and nullable, so the function just works well.

Therefore, we have temporarily added a branch for Spark Connect in the code so that we can create InternalFrame properly to provide more pandas APIs in Spark Connect. If a clear cause is found, we may need to revert it back to its original state.
Bug: DataFrame.to(schema) fails when it contains non-nullable nested field in nullable field
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/23 21:53
Affects Version/s: 3.4.0

{{DataFrame.to(schema)}} fails when it contains non-nullable nested field in nullable field:
{code:scala}
scala> val df = spark.sql("VALUES (1, STRUCT(1 as i)), (NULL, NULL) as t(a, b)")
df: org.apache.spark.sql.DataFrame = [a: int, b: struct<i: int>]
scala> df.printSchema()
root
 |-- a: integer (nullable = true)
 |-- b: struct (nullable = true)
 |    |-- i: integer (nullable = false)

scala> df.to(df.schema)
org.apache.spark.sql.AnalysisException: [NULLABLE_COLUMN_OR_FIELD] Column or field `b`.`i` is nullable while it's required to be non-nullable.
{code}
Bug: Session variables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/23 20:32
Affects Version/s: 3.5.0

Provide a type-safe, engine controlled session variable:

CREATE [ OR REPLACE } TEMPORARY VARIABLE [ IF NOT EXISTS ]var_name  [ type ][ DEFAULT expresion ]

SET {  variable = expression | ( variable [, ...] ) = ( subquery | expression [, ...] )

DROP VARIABLE  [ IF EXISTS ]variable_name
Bug: Structured Streaming with Spark Connect
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/23 22:41
Affects Version/s: 3.5.0

Spark-connect (SPARK-39375) should support structured streaming. 

This Epic covers various tasks for adding streaming support.

The current focus is on {*}Python API{*}, unless otherwise stated, the tasks below are for Python API.
Bug: Add Identifier to the cached RDD node on the Stages page 
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Mar/23 18:39
Affects Version/s: 3.3.2

On the stages page in the Web UI, there is no distinction for which cached RDD is being executed in a particular stage. This Jira aims to add an repeat identifier to distinguish which cached RDD is being executed.
Bug: Incorrect ambiguous column reference error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/23 18:43
Affects Version/s: 3.2.0

val df1 = sc.parallelize(List((1,2,3,4,5),(1,2,3,4,5))).toDF("id","col2","col3","col4", "col5")
val op_cols_same_case = List("id","col2","col3","col4", "col5", "id")
val df2 = df1.select(op_cols_same_case.head, op_cols_same_case.tail: _*)
df2.select("id").show()
 
This query runs fine.
 
But when we change the casing of the op_cols to have mix of upper & lower case ("id" & "ID") it throws an ambiguous col ref error:
 
val df1 = sc.parallelize(List((1,2,3,4,5),(1,2,3,4,5))).toDF("id","col2","col3","col4", "col5")
val op_cols_mixed_case = List("id","col2","col3","col4", "col5", "ID")
val df3 = df1.select(op_cols_mixed_case.head, op_cols_mixed_case.tail: _*)
df3.select("id").show()



org.apache.spark.sql.AnalysisException: Reference 'id' is ambiguous, could be: id, id.

  at org.apache.spark.sql.catalyst.expressions.package$AttributeSeq.resolve(package.scala:363)

  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveChildren(LogicalPlan.scala:112)

  at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$resolveExpressionByPlanChildren$1(Analyzer.scala:1857)

  at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$resolveExpression$2(Analyzer.scala:1787)

  at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:60)

  at org.apache.spark.sql.catalyst.analysis.Analyzer.innerResolve$1(Analyzer.scala:1794)

  at org.apache.spark.sql.catalyst.analysis.Analyzer.resolveExpression(Analyzer.scala:1812)

  at org.apache.spark.sql.catalyst.analysis.Analyzer.resolveExpressionByPlanChildren(Analyzer.scala:1863)

  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$17.$anonfun$applyOrElse$94(Analyzer.scala:1577)

  at org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:193)

  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)

  at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:193)

  at org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:204)

  at org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$3(QueryPlan.scala:209)

  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)

  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)

  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)

  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)

  at scala.collection.TraversableLike.map(TraversableLike.scala:286)

  at scala.collection.TraversableLike.map$(TraversableLike.scala:279)

  at scala.collection.AbstractTraversable.map(Traversable.scala:108)

  at org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:209)

 


Since, Spark is case insensitive, it should work for second case also when we have upper and lower case column names in the column list.

It also works fine in Spark 2.3.
Bug: DataSourceV2 cannot report KeyGroupedPartitioning with multiple keys per partition
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Mar/23 11:14
Affects Version/s: 3.3.0

From Spark 3.0.0 until 3.2.3, a DataSourceV2 could report its partitioning as {{KeyGroupedPartitioning}} via {{SupportsReportPartitioning}}, even if multiple keys belong to a partition.

With SPARK-37377, only if all partitions implement {{HasPartitionKey}}, the partition information reported through {{SupportsReportPartitioning}} is considered by catalyst. But this limits the number of keys per partition to 1.

Spark should continue to support the more general situation of {{KeyGroupedPartitioning}} with multiple keys per partition, like {{HashPartitioning}}.
Bug: approx_percentile produces wrong results for large decimals.
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Mar/23 20:53
Affects Version/s: 2.1.0

In the {{approx_percentile}} expression, Spark casts decimal to double to update the aggregation state ([ApproximatePercentile.scala#L181|https://github.com/apache/spark/blob/933dc0c42f0caf74aaa077fd4f2c2e7208452b9b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala#L181]) and casts the result double back to decimal ([ApproximatePercentile.scala#L206|https://github.com/apache/spark/blob/933dc0c42f0caf74aaa077fd4f2c2e7208452b9b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala#L206]). The precision loss in the casts can make the result decimal out of its precision range. This can lead to the following counter-intuitive results:
{code:sql}
spark-sql> select approx_percentile(col, 0.5) from values (9999999999999999999) as tab(col);
NULL
spark-sql> select approx_percentile(col, 0.5) is null from values (9999999999999999999) as tab(col);
false
spark-sql> select cast(approx_percentile(col, 0.5) as string) from values (9999999999999999999) as tab(col);
10000000000000000000
spark-sql> desc select approx_percentile(col, 0.5) from values (9999999999999999999) as tab(col);
approx_percentile(col, 0.5, 10000)	decimal(19,0) 
{code}
The result is actually not null, so the second query returns false. The first query returns null because the result cannot fit into {{{}decimal(19, 0){}}}.

A suggested fix is to use {{Decimal.changePrecision}} here to ensure the result fits, and really returns a null or throws an exception when the result doesn't fit.
Bug: ClassNotFoundException: scala.math.Ordering$Reverse
Status: Open | Priority: Major | Resolution: 
Created: 21/Mar/23 17:40
Affects Version/s: 3.3.2

Hi,

We are using the spark-mllib_2.12 dependency in a Java project.

We are attempting to upgrade from version 3.3.1 to 3.3.2. This results in unit tests breaking with exception: ClassNotFoundException: scala.math.Ordering$Reverse

A change was made to add the class to the KyroSerializer  https://issues.apache.org/jira/browse/SPARK-42071

scala.math.Ordering$Reverse was introduced int Scala 2.12.12. The maven dependency tree (mvn dependency:tree) shows that spark-mllib_2.12 brings in scala-library version 2.12.8. Therefore, it doesn't contain scala.math.Ordering$Reverse. 

If the scala-library transitive dependency is excluded from the POM and an explicit dependency declared on with version >=2.12.12, the tests will pass.

Should the scala-library version contained in 3.3.2 be upgraded to >=2.12.12?
Bug: Add Identifier to the InMemoryTableScan node on the SQL page
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/23 00:51
Affects Version/s: 3.3.2

On the SQL page in the Web UI, there is no distinction for which InMemoryTableScan is being used at a specific point in the DAG. This Jira aims to add a repeat identifier to distinguish which InMemoryTableScan is being used at a certain location.
Bug: Support INSERT INTO by name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/23 17:13
Affects Version/s: 3.4.0

In some use cases, users have incoming dataframes with fixed column names which might differ from the canonical order. Currently there's no way to handle this easily through the INSERT INTO API - the user has to make sure the columns are in the right order as they would when inserting a tuple. We should add an optional BY NAME clause, such that:

INSERT INTO tgt BY NAME <query>



takes each column of <query> and inserts it into the column in `tgt` which has the same name according to the configured `resolver` logic.
Bug: Allow access to stopped streaming queries
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 29/Mar/23 06:12
Affects Version/s: 3.5.0

In Spark connect, when a query is stopped, the client can't access it anymore. That implies they can not fetch any information about the query after that. 

The server might have to cache the queries for some time (upto the time the session is closed).
Bug: Upgrade Hadoop to 3.3.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/23 03:58
Affects Version/s: 3.5.0

* [https://hadoop.apache.org/docs/r3.3.5/hadoop-project-dist/hadoop-common/release/3.3.5/CHANGELOG.3.3.5.html]
 * [https://hadoop.apache.org/docs/r3.3.5/hadoop-project-dist/hadoop-common/release/3.3.5/RELEASENOTES.3.3.5.html]
 * https://hadoop.apache.org/docs/r3.3.5/
Bug: Cast from string to date and date to string say timezone is needed, but it is not used
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/23 15:21
Affects Version/s: 3.2.0

This is really minor but SPARK-35581 removed the need for a timezone when casting from a `StringType` to a `DateType`, but the patch didn't update the `needsTimeZone` function to indicate that it was not longer required.

Currently Casting from a DateType to a StringType also says that it needs the timezone, but it only uses the `DateFormatter` with it's default parameters that do not use the time zone at all.

I think this can be fixed with just a two line change.
Bug: Fix the problem of incomplete creation of subdirectories in push merged localDir
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 08:50
Affects Version/s: 3.3.2

After we massively enabled push-based shuffle in our production environment, we found some warn messages appearing in the server-side log messages.

the warning log like:

ShuffleBlockPusher: Pushing block shufflePush_3_0_5352_935 to BlockManagerId(shuffle-push-merger, zw06-data-hdp-dn08251.mt, 7337, None) failed.
java.lang.RuntimeException: java.lang.RuntimeException: Cannot initialize merged shuffle partition for appId application_1671244879475_44020960 shuffleId 3 shuffleMergeId 0 reduceId 935.

After investigation, we identified the triggering mechanism of the bug。

The driver requested two different containers on the same physical machine. During the creation of the 'push-merged' directory in the first container (container_1), the mergeDir was created first, then the subDir were created based on the value of the "spark.diskStore.subDirectories" parameter. However, the resources of container_1 were preempted during the creation of the sub-directories, resulting in subDir not being created (only part of it was created ). As the mergeDir still existed, the second container (container_2) was unable to create further subDir (as it assumed that all directories had already been created).
Bug: Fix the bug that pushdown offset or paging is invalid for some built-in dialect 
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 10/Mar/23 04:54
Affects Version/s: 3.4.0

Currently, the default pushdown offset like OFFSET n. But some built-in dialect doesn't support the syntax. So when the Spark pushdown offset into these databases, them  throwing errors.
Bug: Block Arrow Python UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/23 02:18
Affects Version/s: 3.4.0

Considering the upcoming improvements on the result inconsistencies between traditional Pickled Python UDFs and Arrow-optimized Python UDFs, we'd better block the feature, otherwise, users who try out the feature will expect behavior changes in the next release.

In addition, since Spark Connect Python Client(SCPC) has been introduced in Spark 3.4, we'd better ensure the feature is ready in both vanilla PySpark and SCPC at the same time for compatibility.
Bug: Artifact transfer from Scala/JVM client to Server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/23 15:37
Affects Version/s: 3.4.0

In the decoupled client-server architecture of Spark Connect, a remote client may use a local JAR or a new class in their UDF that may not be present on the server. To handle these cases of missing "artifacts", we need to implement a mechanism to transfer artifacts from the client side over to the server side as per the protocol defined in https://github.com/apache/spark/pull/40147
Bug: The partition of result data frame of join is always 1
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 11/Mar/23 14:48
Affects Version/s: 3.3.2

I am using pyspark. The partition of result data frame of join is always 1.

Here is my code from https://stackoverflow.com/questions/51876281/is-partitioning-retained-after-a-spark-sql-join

 

print(spark.version)

def example_shuffle_partitions(data_partitions=10, shuffle_partitions=4):
    spark.conf.set("spark.sql.shuffle.partitions", shuffle_partitions)
    spark.sql("SET spark.sql.autoBroadcastJoinThreshold=-1")
    df1 = spark.range(1, 1000).repartition(data_partitions)
    df2 = spark.range(1, 2000).repartition(data_partitions)
    df3 = spark.range(1, 3000).repartition(data_partitions)

    print("Data partitions is: {}. Shuffle partitions is {}".format(data_partitions, shuffle_partitions))
    print("Data partitions before join: {}".format(df1.rdd.getNumPartitions()))

    df = (df1.join(df2, df1.id == df2.id)
          .join(df3, df1.id == df3.id))

    print("Data partitions after join : {}".format(df.rdd.getNumPartitions()))

example_shuffle_partitions()

 


In Spark 3.0.3, it prints out:
3.0.3
Data partitions is: 10. Shuffle partitions is 4
Data partitions before join: 10
Data partitions after join : 4


However, it prints out the following in the latest 3.3.2
3.3.2
Data partitions is: 10. Shuffle partitions is 4
Data partitions before join: 10
Data partitions after join : 1
Bug: Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/23 00:05
Affects Version/s: 3.4.0

Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort

We have seen some cases where the task exits as cancelled/failed which triggers the abort in the task completion listener for HDFSStateStoreProvider. As part of this, we cancel the backing stream and close the compressed stream. However, different stores such as Azure blob store could throw exceptions which are not caught in the current path, leading to job failures. This change proposes to fix this issue.
Bug: Upgrade google Tink to 1.9.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 08:07
Affects Version/s: 3.5.0

[SNYK-JAVA-COMGOOGLEPROTOBUF-3040284|https://security.snyk.io/vuln/SNYK-JAVA-COMGOOGLEPROTOBUF-3040284]

[SNYK-JAVA-COMGOOGLEPROTOBUF-3167772|https://security.snyk.io/vuln/SNYK-JAVA-COMGOOGLEPROTOBUF-3167772]
Bug: Jackson 2.15
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 19/Mar/23 10:49
Affects Version/s: 3.4.1

I'm not advocating for an upgrade to [Jackson 2.15|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15]. 2.15.0-rc1 has just been released and 2.15.0 should be out soon.

There are some security focused enhancements including a new class called StreamReadConstraints. The defaults on [StreamReadConstraints|https://javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.15.0-rc1/com/fasterxml/jackson/core/StreamReadConstraints.html] are pretty high but it is not inconceivable that some Spark users might need to relax them. Parsing large strings as numbers is sub-quadratic, thus the default limit of 1000 chars or bytes (depending on input context).

When the Spark team consider upgrading to Jackson 2.15 or above, you might also want to consider adding some way for users to configure the StreamReadConstraints.
Bug: Session management support streaming connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/23 23:00
Affects Version/s: 3.5.0

Add session support for streaming jobs. 

E.g. a session should stay alive when a streaming job is alive. 

It might differ more complex scenarios like what happens when client loses track of the session. Such semantics would be handled as part of session semantics across Spark Connect (including streaming).
Bug: can not analyze window exp on sub query
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/23 09:59
Affects Version/s: 3.2.1

 

CREATE TABLE test_noindex_table(`name` STRING,`age` INT,`city` STRING) PARTITIONED BY (`date` STRING);

 

SELECT
    *
FROM
(
    SELECT *, COUNT(1) OVER itr AS grp_size
    FROM test_noindex_table 
    WINDOW itr AS (PARTITION BY city)
) tbl
WINDOW itr2 AS (PARTITION BY
    city
)
 
Window specification itr is not defined in the WINDOW clause.
  !image-2023-03-20-18-00-40-578.png|width=560,height=361!
Bug: Support to find and transfer client-side REPL classfiles to server as artifacts  
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/23 22:43
Affects Version/s: 3.4.0

To run UDFs which are defined on the client side REPL, we require a mechanism that can find the local REPL classfiles and then utilise the mechanism from https://issues.apache.org/jira/browse/SPARK-42653 to transfer them to the server as artifacts.
Bug: Fix `default_session` to work properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/23 16:13
Affects Version/s: 3.5.0

Currently, default_session is not working properly in Spark Connect as below:
{code:java}
>>> spark = default_session()
>>> spark.conf.set("default_index_type", "sequence")
>>> spark.conf.get("default_index_type")
'sequence'
>>>
>>> spark = default_session()
>>> spark.conf.get("default_index_type")
Traceback (most recent call last):
...
pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.util.NoSuchElementException) default_index_type
{code}
It should work as expected in regular PySpark as below:
{code:java}
>>> spark = default_session()
>>> spark.conf.set("default_index_type", "sequence")
>>> spark.conf.get("default_index_type")
'sequence'
>>>
>>> spark = default_session()
>>> spark.conf.get("default_index_type")
'sequence'{code}
Bug: Fix test_createDataFrame_with_single_data_type.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/23 21:23
Affects Version/s: 3.4.0

PySpark raises an exception when:

{code:python}
>>> spark.createDataFrame(pd.DataFrame({"a": [1]}), schema="int").collect()
Traceback (most recent call last):
...
TypeError: field value: IntegerType() can not accept object (1,) in type <class 'tuple'>
{code}

whereas Spark Connect doesn't.
Bug: Fix createDataFrame from pandas with map type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/23 21:10
Affects Version/s: 3.4.0

{code:python}
>>> import pandas as pd
>>>
>>> map_data = [{"a": 1}, {"b": 2, "c": 3}, {}, None, {"d": None}]
>>> pdf = pd.DataFrame({"id": [0, 1, 2, 3, 4], "m": map_data})
>>> schema = "id long, m map<string, long>"
>>> spark.createDataFrame(pdf, schema=schema)
Traceback (most recent call last):
...
pyspark.errors.exceptions.connect.AnalysisException: [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `col_1` is of type "STRUCT<col_0: BIGINT, col_1: BIGINT, col_2: BIGINT, col_3: VOID>" while it's required to be "MAP<STRING, BIGINT>".
{code}
Bug: Execution plan error, unable to obtain desired results
Status: Open | Priority: Major | Resolution: 
Created: 28/Mar/23 10:19
Affects Version/s: 3.2.0

A jar is packaged using SparkSession to submit Spark SQL：
{code:java}
//SparkSession.builder().appName(args(0)).config("spark.sql.crossJoin.enabled", true).enableHiveSupport().getOrCreate() spark.sql(arg(1)) {code}
Execute the following SQL fragment：
{code:java}
//INSERT INTO gjdw.aa partition(dt='20230327')
SELECT t1.mandt,
       t1.pur_no,
       t1.pur_item,
       t1.pur_comp_code,
       t1.pur_pur_org,
       t1.zzcoca,
       t1.zzycgdd
FROM
  (SELECT *
   FROM gjdw.aa
   WHERE dt=from_unixtime(unix_timestamp(date_add(from_unixtime(unix_timestamp('20230327','yyyymmdd'),'yyyy-mm-dd'),-1),'yyyy-mm-dd'),'yyyymmdd')) t1
LEFT JOIN
  (SELECT *
   FROM gjdw.aa
   WHERE dt='20230327') t ON t.pur_no = t1.pur_no
AND t.pur_item = t1.pur_item
WHERE (t.pur_no = ''
       AND t.pur_item = ''
       OR (t.pur_no IS NULL
           AND t.pur_item IS NULL)) {code}
 

Strangely, I didn't get the desired result. There was data in the table, and the correct value should have data inserted. However, there was no data output, and there was no task error message for the job. This occurred in the execution plan

!image-2023-04-10-11-39-06-501.png!

!image-2023-04-10-11-39-33-658.png!
Bug: Add analysed logical mode in org.apache.spark.sql.execution.ExplainMode
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 20/Mar/23 03:55
Affects Version/s: 3.2.0

We have a service that submits spark sql jobs to a spark cluster .

we want to validate the sql query before submitting the job . We are currently using df.explain(extended=true) which generates parsed , analysed , optimised logical plan and physical plan . 

But generating  optimised logical plan  sometimes takes more time for e.g if you have applied a filter on a partitioned column , spark will list all directories and take the required ones . 

For our query validation purpose this doesnt make sense and it would be great if there is a explain mode that will only print the parsed and analysed logical plans only