av9ash/GitBugs huggingface dataset:
Bug: Default to ParquetOutputCommitter even after configuring s3a committer as "partitioned"
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 06/Mar/20 15:42
Affects Version/s: 2.4.5

My program logs says it uses ParquetOutputCommitter when I use _*"Parquet"*_ even after I configure to use "PartitionedStagingCommitter" with the following configuration:
 * sparkSession.conf().set("spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a", "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory");
 * sparkSession.conf().set("fs.s3a.committer.name", "partitioned");
 * sparkSession.conf().set("fs.s3a.committer.staging.conflict-mode", "append");
 * sparkSession.conf().set("spark.hadoop.parquet.mergeSchema", "false");
 * sparkSession.conf().set("spark.hadoop.parquet.enable.summary-metadata", false);

Application logs stacktrace:

20/03/06 10:15:17 INFO ParquetFileFormat: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter
20/03/06 10:15:17 INFO FileOutputCommitter: File Output Committer Algorithm version is 2
20/03/06 10:15:17 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
20/03/06 10:15:17 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
20/03/06 10:15:17 INFO FileOutputCommitter: File Output Committer Algorithm version is 2
20/03/06 10:15:17 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
20/03/06 10:15:17 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter

But when I use _*ORC*_ as the file format, with the same configuration as above it correctly pick "PartitionedStagingCommitter":
20/03/05 11:51:14 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
20/03/05 11:51:14 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
20/03/05 11:51:14 INFO AbstractS3ACommitterFactory: Using committer partitioned to output data to s3a:************
20/03/05 11:51:14 INFO AbstractS3ACommitterFactory: Using Commmitter PartitionedStagingCommitter**********

So I am wondering why Parquet and ORC has different behavior ?
How can I use PartitionedStagingCommitter instead of ParquetOutputCommitter?

I started this because when I was trying to save data to S3 directly with partitionBy() two columns -  I was getting  file not found exceptions intermittently.  
So how could I avoid this issue with *Parquet  using Spark to S3 using s3A without s3aGuard?*
Bug: Support DDL "SHOW VIEWS"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/20 04:03
Affects Version/s: 3.0.0

It is nice to have a `SHOW VIEWS` command similar to Hive ([https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowViews]).

Although this is adding a new feature, this resolved the regression bug in the below.

According to the report, [https://github.com/apache/spark/pull/27897#issuecomment-602519201],
{quote}Tableau and PowerBI, which use the Simba ODBC driver, do not work with Spark 3.0 with Hive 2.3.

When we switch to Hive 2.3 profile, Simba driver assumes that {{SHOW VIEWS}} is supported, and issues it, failing the {{SQLGetTables}} call when it fails.
{quote}
Bug: Support 'SHOW views' command
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 04/Mar/20 05:13
Affects Version/s: 3.1.0

{noformat}
spark-sql> show views;
Error in query:
missing 'FUNCTIONS' at '<EOF>'(line 1, pos 10)

== SQL ==
show views
----------^^^

spark-sql> show views in default;
Error in query:
missing 'FUNCTIONS' at 'in'(line 1, pos 11)

== SQL ==
show views in default
-----------^^^
{noformat}

Hive 3.1 support this command:
{noformat}
hive> select version();
OK
3.1.2 r8190d2be7b7165effa62bd21b7d60ef81fb0e4af
Time taken: 4.874 seconds, Fetched: 1 row(s)
hive> show views in default;
OK
Time taken: 0.01 seconds
hive> show views;
OK
Time taken: 0.013 seconds
hive> show views in default;
OK
Time taken: 0.013 seconds
{noformat}

*PowerBI* also need this command:
Bug: Backward Compatibility for Parsing and Formatting Datetime
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/20 02:41
Affects Version/s: 3.0.0

*Background*

In Spark version 2.4 and earlier, datetime parsing, formatting and conversion are performed by using the hybrid calendar ([Julian + Gregorian|https://docs.oracle.com/javase/7/docs/api/java/util/GregorianCalendar.html]). 

Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes (the java.time packages that are based on [ISO chronology|https://docs.oracle.com/javase/8/docs/api/java/time/chrono/IsoChronology.html] ).

The switching job is completed in SPARK-26651. 

 

*Problem*

Switching to Java 8 datetime API breaks the backward compatibility of Spark 2.4 and earlier when parsing datetime. Spark need its own patters definition on datetime parsing and formatting.

 

*Solution*

To avoid unexpected result changes after the underlying datetime API switch, we propose the following solution. 
 * Introduce the fallback mechanism: when the Java 8-based parser fails, we need to detect these behavior differences by falling back to the legacy parser, and fail with a user-friendly error message to tell users what gets changed and how to fix the pattern.

 * Document the Spark’s datetime patterns: The date-time formatter of Spark is decoupled with the Java patterns. The Spark’s patterns are mainly based on the [Java 7’s pattern|https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html] (for better backward compatibility) with the customized logic (caused by the breaking changes between [Java 7|https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html] and [Java 8|https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html] pattern string). Below are the customized rules:

||Pattern||Java 7||Java 8|| Example||Rule||
|u|Day number of week (1 = Monday, ..., 7 = Sunday)|Year (Different with y, u accept a negative value to represent BC, while y should be used together with G to do the same thing.)|!image-2020-03-04-10-54-05-208.png!  |Substitute ‘u’ to ‘e’ and use Java 8 parser to parse the string. If parsable, return the result; otherwise, fall back to ‘u’, and then use the legacy Java 7 parser to parse. When it is successfully parsed, throw an exception and ask users to change the pattern strings or turn on the legacy mode; otherwise, return NULL as what Spark 2.4 does.|
| z| General time zone which also accepts
 [RFC 822 time zones|#rfc822timezone]]|Only accept time-zone name, e.g. Pacific Standard Time; PST|!image-2020-03-04-10-54-13-238.png!  |The semantics of ‘z’ are different between Java 7 and Java 8. Here, Spark 3.0 follows the semantics of Java 8. 
 Use Java 8 to parse the string. If parsable, return the result; otherwise, use the legacy Java 7 parser to parse. When it is successfully parsed, throw an exception and ask users to change the pattern strings or turn on the legacy mode; otherwise, return NULL as what Spark 2.4 does.|
Bug: Support is_json function
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Mar/20 09:51
Affects Version/s: 3.1.0

This function will allow users to verify whether the given string is valid JSON or not. It returns `true` for valid JSON and `false` for invalid JSON. `NULL` is returned for `NULL` input.

DBMSs supporting this functions are :
 * MySQL
 * SQL Server
 * Sqlite
 * MariaDB
 * Amazon Redshift
 * IBM Db2
Bug: Improve SizeEstimator for AppendOnlyMap
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Mar/20 09:54
Affects Version/s: 2.3.2

Currently, spark's memory management depends on the size estimation for execution and storage. 
In our real cluster, users always meet the issue OOM due to the inaccurate size estimation for ` AppendOnlyMap`, that's because spark stores KV in an Array[AnyRef] in `AppendOnlyMap` for memory locality,  and this value can be CompactBuffer[_] or Array[CompactBuffer[_]] for transformation like cogroup/join/groupBy, but current `SizeEstimator` will still treat this special array as an normal array, so in many cases, we noticed a great bias between the estimated size and the acutal memory consuption. 
So we improved this in xiaomi:
1. Improve the estimation for AppendOnlyMap when the value type is CompactBuffer
2. Respect jvm gc stats to decide whether to spilling when doing sort/agg

In this jira, I propose to solve the first part which is improving the estimation for `AppendOnlyMap` when the value type is CompactBuffer
Bug: Expose basic write metrics for InsertIntoDataSourceCommand
Status: In Progress | Priority: Major | Resolution: 
Created: 14/Mar/20 07:15
Affects Version/s: 3.1.0

Spark provides interface `InsertableRelation` and the `InsertIntoDataSourceCommand` to delegate the inserting processing to a data source. Unlike `DataWritingCommand`, the metrics in InsertIntoDataSourceCommand is empty and has no chance to update. So we cannot get "number of written files" or "number of output rows" from its metrics.

For example, if a table is a Spark parquet table. We can get the writing metrics by:
{code}
val df = sql("INSERT INTO TABLE test_table SELECT 1, 'a'")
val numFiles = df.queryExecution.sparkPlan.metrics("numFiles").value
{code}
But if it is a Delta table, we cannot.
Bug: Extend FairScheduler to support pool level resource isolation
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Mar/20 10:00
Affects Version/s: 3.1.0

Currently, spark only provided two types of scheduler: FIFO & FAIR, but in sql high-concurrency scenarios, a few of drawbacks are exposed.

FIFO: it can easily causing congestion when large sql query occupies all the resources

FAIR: the taskSets of one pool may occupies all the resource due to there are no hard limit on the maximum usage for each pool.  this case may be frequently met under high workloads.

So we propose to add a maxShare argument for FairScheduler to control the maximum running tasks for each pool.

One thing that needs our attention is that we should handle it well to make the `ExecutorAllocationManager` can release resources:
 e.g. Suppose we got 100 executors, if the tasks are scheduled on all executors with max concurrency 50, there are cases that the executors may not idle, and can not be released.

One idea is to bind those executors to each pool, then we only schedule tasks on executors of the pool which it belongs to.
Bug: PySpark job not killing Spark Daemon processes after the executor is killed due to OOM
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 13/Mar/20 15:27
Affects Version/s: 2.4.5

{code:java}
2020-03-10 10:15:00,257 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 327523 for container-id container_e25_1583
485217113_0347_01_000042: 1.9 GB of 2 GB physical memory used; 39.5 GB of 4.2 GB virtual memory used
2020-03-10 10:15:05,135 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 327523 for container-id container_e25_1583
485217113_0347_01_000042: 3.6 GB of 2 GB physical memory used; 41.1 GB of 4.2 GB virtual memory used
2020-03-10 10:15:05,136 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree for container: container_e25_1583485217113_0347_01_000042
 has processes older than 1 iteration running over the configured limit. Limit=2147483648, current usage = 3915513856
2020-03-10 10:15:05,136 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=327523,containerID=container_e25_1583485217113_0347_01_
000042] is running beyond physical memory limits. Current usage: 3.6 GB of 2 GB physical memory used; 41.1 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_e25_1583485217113_0347_01_000042 :
        |- 327535 327523 327523 327523 (java) 1611 111 4044427264 172306 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/bin/java -server -Xmx1024m -Djava.io.tmpdir=/data/s
cratch/yarn/usercache/u689299/appcache/application_1583485217113_0347/container_e25_1583485217113_0347_01_000042/tmp -Dspark.ssl.trustStore=/opt/mapr/conf/ssl_truststore -Dspark.authenticat
e.enableSaslEncryption=true -Dspark.driver.port=40653 -Dspark.network.timeout=7200 -Dspark.ssl.keyStore=/opt/mapr/conf/ssl_keystore -Dspark.network.sasl.serverAlwaysEncrypt=true -Dspark.ssl
.enabled=true -Dspark.ssl.protocol=TLSv1.2 -Dspark.ssl.fs.enabled=true -Dspark.ssl.ui.enabled=false -Dspark.authenticate=true -Dspark.yarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.7.
0/logs/userlogs/application_1583485217113_0347/container_e25_1583485217113_0347_01_000042 -XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler@bd02slse0201.wellsfargo.com:40653 --executor-id 40 --hostname bd02slsc0519.wellsfargo.com --cores 1 --app-id application_1583485217113_0347 --user-class-path
file:/data/scratch/yarn/usercache/u689299/appcache/application_1583485217113_0347/container_e25_1583485217113_0347_01_000042/__app__.jar
{code}
 

 

After that, there are lots of pyspark.daemon process left.
 eg:
 /apps/anaconda3-5.3.0/bin/python -m pyspark.daemon
Bug: Dropna doesn't work for struct columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 22:56
Affects Version/s: 2.4.5

Dropna using a subset with a column from a struct drops the entire data frame.
{code:python}
import pyspark.sql.functions as F

df = spark.createDataFrame([(5, 80, 'Alice'), (10, None, 'Bob'), (15, 80, None)], schema=['age', 'height', 'name'])
df.show()
+---+------+-----+
|age|height| name|
+---+------+-----+
|  5|    80|Alice|
| 10|  null|  Bob|
| 15|    80| null|
+---+------+-----+

# this works just fine
df.dropna(subset=['name']).show()
+---+------+-----+
|age|height| name|
+---+------+-----+
|  5|    80|Alice|
| 10|  null|  Bob|
+---+------+-----+

# now add a struct column
df_with_struct = df.withColumn('struct_col', F.struct('age', 'height', 'name'))
df_with_struct.show(truncate=False)
+---+------+-----+--------------+
|age|height|name |struct_col    |
+---+------+-----+--------------+
|5  |80    |Alice|[5, 80, Alice]|
|10 |null  |Bob  |[10,, Bob]    |
|15 |80    |null |[15, 80,]     |
+---+------+-----+--------------+

# now dropna drops the whole dataframe when you use struct_col
df_with_struct.dropna(subset=['struct_col.name']).show(truncate=False)
+---+------+----+----------+
|age|height|name|struct_col|
+---+------+----+----------+
+---+------+----+----------+
{code}
 I've tested the above code in Spark 2.4.4 with python 3.7.4 and Spark 2.3.1 with python 3.6.8 and in both, the result looks like:
{code:python}
df_with_struct.dropna(subset=['struct_col.name']).show(truncate=False)
+---+------+-----+--------------+
|age|height|name |struct_col    |
+---+------+-----+--------------+
|5  |80    |Alice|[5, 80, Alice]|
|10 |null  |Bob  |[10,, Bob]    |
+---+------+-----+--------------+
{code}
Bug: flatten the result dataframe of tests in stat
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/20 09:23
Affects Version/s: 3.1.0

{code:java}
 scala> import org.apache.spark.ml.linalg.{Vector, Vectors}
import org.apache.spark.ml.linalg.{Vector, Vectors}scala> import org.apache.spark.ml.stat.ChiSquareTest
import org.apache.spark.ml.stat.ChiSquareTestscala>     val data = Seq(
     |       (0.0, Vectors.dense(0.5, 10.0)),
     |       (0.0, Vectors.dense(1.5, 20.0)),
     |       (1.0, Vectors.dense(1.5, 30.0)),
     |       (0.0, Vectors.dense(3.5, 30.0)),
     |       (0.0, Vectors.dense(3.5, 40.0)),
     |       (1.0, Vectors.dense(3.5, 40.0))
     |     )
data: Seq[(Double, org.apache.spark.ml.linalg.Vector)] = List((0.0,[0.5,10.0]), (0.0,[1.5,20.0]), (1.0,[1.5,30.0]), (0.0,[3.5,30.0]), (0.0,[3.5,40.0]), (1.0,[3.5,40.0]))scala> scala> scala> val df = data.toDF("label", "features")
df: org.apache.spark.sql.DataFrame = [label: double, features: vector]scala>     val chi = ChiSquareTest.test(df, "features", "label")
chi: org.apache.spark.sql.DataFrame = [pValues: vector, degreesOfFreedom: array<int> ... 1 more field]scala> chi.show
+--------------------+----------------+----------+
|             pValues|degreesOfFreedom|statistics|
+--------------------+----------------+----------+
|[0.68728927879097...|          [2, 3]|[0.75,1.5]|
+--------------------+----------------+----------+{code}
 

Current impls of {{ChiSquareTest}}, {{ANOVATest}}, {{FValueTest}}, {{Correlation}} all return a df only containing one row.

I think this is quite hard to use, suppose we have a dataset with dim=1000, the only operation we can deal with the test result is to collect it by {{head()}} or {{first(), and then use it in the driver.}}

{{While what I really want to do is filtering the df like pValue>0.1}} or {{corr<0.5}}, *So I suggest to flatten the output df in those tests.*

 

{{note: {{ANOVATest}}{{}}}} and\{{FValueTest}} are newly added in 3.1.0, but {{{{ChiSquareTest}}}} and {{{{Correlation}}}} were here for a long time.
Bug: catalog plugin API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/20 11:35
Affects Version/s: 3.0.0

Details please see the SPIP doc: https://docs.google.com/document/d/1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo/edit#heading=h.m45webtwxf2d

This will bring multi-catalog support to Spark and allow external catalog implementations.
Bug: Remove support for 'e'/'c' as datetime pattern charactar 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/20 17:00
Affects Version/s: 3.0.0

The meaning of 'u' was day number of week in SimpleDateFormat, it was changed to year in DateTimeFormatter.  So we keep the old meaning of 'u' by substituting 'u' to 'e' internally and use DateTimeFormatter to parse the pattern string. In DateTimeFormatter, the 'e' and 'c' also represents day-of-week, we should mark them as illegal pattern character to stay the same as before.
Bug: Parsing seconds fraction with variable length for timestamp
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Mar/20 18:00
Affects Version/s: 3.1.0

This JIRA is to support parsing timestamp values with variable length second fraction parts.

e.g. 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]' can parse timestamp with 0~6 digit-length second fraction but fail >=7


{code:java}
select to_timestamp(v, 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]') from values
 ('2019-10-06 10:11:12.'),
 ('2019-10-06 10:11:12.0'),
 ('2019-10-06 10:11:12.1'),
 ('2019-10-06 10:11:12.12'),
 ('2019-10-06 10:11:12.123UTC'),
 ('2019-10-06 10:11:12.1234'),
 ('2019-10-06 10:11:12.12345CST'),
 ('2019-10-06 10:11:12.123456PST') t(v)
2019-10-06 03:11:12.123
2019-10-06 08:11:12.12345
2019-10-06 10:11:12
2019-10-06 10:11:12
2019-10-06 10:11:12.1
2019-10-06 10:11:12.12
2019-10-06 10:11:12.1234
2019-10-06 10:11:12.123456

select to_timestamp('2019-10-06 10:11:12.1234567PST', 'yyyy-MM-dd HH:mm:ss.SSSSSS[zzz]')
NULL
{code}

Since 3.0, we use java 8 time API to parse and format timestamp values. when we create the DateTimeFormatter, we use appendPattern to create the build first, where the 'S..S' part will be parsed to a fixed-length(= 'S..S'.length). This fits the formatting part but too strict for the parsing part because the trailing zeros are very likely to be truncated.
Bug: Extend Support for useful JSON Functions
Status: Open | Priority: Major | Resolution: 
Created: 10/Mar/20 09:13
Affects Version/s: 3.1.0

Currently, Spark only supports few functions for JSON. There are many other common utility functions which are supported by other popular DBMSs. Supporting these functions will make it easier for prospective users. Also some functions like `json_array_length` , `json_object_keys` are more intuitive and naive users life would be much simpler.

I have added some JSON functions on which I am working on.
Bug: Mark Spark streaming as deprecated and add warnings.
Status: Open | Priority: Major | Resolution: 
Created: 02/Mar/20 08:22
Affects Version/s: 3.0.0

It is noticed that some of the users of Spark streaming do not immediately realise that it is a deprecated component and it would be scary, if they end up with it in production. Now that we are in a position to release about Spark 3.0.0, may be we should discuss - should the spark streaming carry an explicit notice? That it is not under active development.
Bug: Support json_array_length function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/20 10:51
Affects Version/s: 3.1.0

At the moment we don't support json_array_length function in spark.

This function is supported by

a.) PostgreSQL -> [https://www.postgresql.org/docs/9.3/functions-json.html]

b.) Presto -> [https://prestodb.io/docs/current/functions/json.html]

c.) redshift -> [https://docs.aws.amazon.com/redshift/latest/dg/JSON_ARRAY_LENGTH.html]

 

This allows naive users to directly get array length with a well defined json function.
Bug: numOutputRows shows value from last micro batch when there is no new data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/20 19:39
Affects Version/s: 3.0.0

In Structured Streaming, we provide progress updates every 10 seconds when a stream doesn't have any new data upstream. When providing this progress though, we zero out the input information but not the output information.
Bug: Transforming Hive simple UDF (using JAR) expression may incur CNFE in later evaluation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/20 07:42
Affects Version/s: 2.4.5

In SPARK-26560, we ensured that Hive UDF using JAR is executed regardless of current thread context classloader.

[~cloud_fan] pointed out another potential issue in post-review of SPARK-26560 - quoting the comment:

{quote}
Found a potential problem: here we call HiveSimpleUDF.dateType (which is a lazy val), to force to load the class with the corrected class loader.

However, if the expression gets transformed later, which copies HiveSimpleUDF, then calling HiveSimpleUDF.dataType will re-trigger the class loading, and at that time there is no guarantee that the corrected classloader is used.

I think we should materialize the loaded class in HiveSimpleUDF.
{quote}

This JIRA issue is to track the effort of verifying the potential issue and fixing the issue.
Bug: Add interval value support for extract expression as source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/20 09:24
Affects Version/s: 3.1.0

{code:java}
<extract expression> ::= EXTRACT <left paren> <extract field> FROM <extract source> <right paren>

<extract source> ::= <datetime value expression> | <interval value expression>
{code}

We now only support datetime values as extract source for expression but it's alternative function `date_part` supports both datetime and interval.

For ANSI compliance and the consistency between extract and `date_part`, we support intervals for extract expressions.
Bug: Support views in both SHOW CREATE TABLE and SHOW CREATE TABLE AS SERDE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/20 06:32
Affects Version/s: 3.1.0

For now {{SHOW CREATE TABLE}} command doesn't support views, but {{SHOW CREATE TABLE AS SERDE}} supports it. Since the views syntax are the same between Hive DDL and Spark DDL, we should be able to support views in both two commands.