av9ash/GitBugs huggingface dataset:
Bug: Inconsistent missing values handling in Pandas on Spark to_json
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jan/22 20:37
Affects Version/s: 3.2.1

If {{ps.DataFrame.to_json}} is called without {{path}} argument, missing values are written explicitly 

{code:python}
import pandas as pd
import pyspark.pandas as ps

pdf = pd.DataFrame({"id": [1, 2, 3], "value": [None, 3, None]})
psf = ps.from_pandas(pdf)
psf.to_json()
## '[{"id":1,"value":null},{"id":2,"value":3.0},{"id":3,"value":null}]'
{code:python}

This behavior is consistent with Pandas:

{code:python}
pdf.to_json()
## '{"id":{"0":1,"1":2,"2":3},"value":{"0":null,"1":3.0,"2":null}}'
{code}

However, if {{path}} is provided, missing values are omitted by default:


{code:python}
import tempfile

path = tempfile.mktemp()
psf.to_json(path)

spark.read.text(path).show()
## +--------------------+
## |               value|
## +--------------------+
## |{"id":2,"value":3.0}|
## |            {"id":3}|
## |            {"id":1}|
## +--------------------+
{code}


We should set {{ignoreNullFields}} for Pandas API, to be `False` by default, so both cases handle missing values in the same way.


{code:python}
psf.to_json(path, ignoreNullFields=False)
spark.read.text(path).show(truncate=False)


## +---------------------+
## |value                |
## +---------------------+
## |{"id":3,"value":null}|
## |{"id":1,"value":null}|
## |{"id":2,"value":3.0} |
## +---------------------+
{code}
Bug: The ConfigMap is assigned to incorrect namespace
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 15/Jan/22 10:42
Affects Version/s: 3.3.0

SPARK-37713 added the namespace to ConfigMap. However, the confFileMap does not have namespace value, so it assigns the ConfigMap to "default" namespace. When we submit spark application in non-default namespace, it causes error since it can't find the configmap in non-default namespace
Bug: Persist() and cache() methods shorten LogicalPlan
Status: Open | Priority: Major | Resolution: 
Created: 29/Jan/22 23:15
Affects Version/s: 3.0.1

A case has been noticed when calling the toJSON method for LogicalPlan truncates the plan. In case the DataFrame was cached before. Demonstration of the case:

[Project in Github|https://github.com/MinorityMeaning/CutPlanDataFrame]
Bug: Improve the performance of DS V2 aggregate push-down
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Jan/22 01:58
Affects Version/s: 3.3.0

Currently, the build-in scan builder that implement SupportsPushDownAggregates has itself's config to control aggregate push-down.
We can advance the judgement of the config which controls aggregate push-down.
Bug: Push-based shuffle's internal implementation details should not be exposed as API
Status: Open | Priority: Major | Resolution: 
Created: 29/Jan/22 01:45
Affects Version/s: 3.2.1

Currently the changes added in Dependency.scala for Push-based shuffle are all exposed as Developer API. For eg: `newShuffleMergeState`, `setMergerLocs`, `shuffleMergeFinalized` etc Some of these might be just internal implementation details which should not be leaked. This JIRA Is to appropriately mark them private as needed.

cc [~mridulm80][~mshen][~csingh][~Ngone51][~tgraves]
Bug: From Scala/Java API, higher-level functions (like exists) produce wrong expressions when nested
Status: Open | Priority: Major | Resolution: 
Created: 26/Jan/22 15:43
Affects Version/s: 3.1.1

Consider this code in Scala:
{code:scala}
case class DemoSubRow(a: Int, b: Array[Int])
case class DemoRow(elems: Array[DemoSubRow])
dataFrame.withColumn(
    "goodElems",
    filter(elems, x -> exists(x.getField("b"), y -> x.getField("a") == y)
{code}
One could expect that is would be equivalent to
{code:sql}
SELECT *, filter(elems, x -> exists(x.b, y -> x.a == y)) AS goodElems
    FROM dataFrame
{code}
However, it's not. If you look into {{org.apache.spark.sql.functions}} object, you'll see that private method:
{code:scala}
private def createLambda(f: Column => Column) = {
  val x = UnresolvedNamedLambdaVariable(Seq("x"))
  val function = f(Column(x)).expr
  LambdaFunction(function, Seq(x))
}
{code}
If you look closely you'll see that column that is passed into the lambda is always *unresolved* variable {{{}x{}}}. Because of that, column from Scala sample above is seen as:
{code:sql}
… filter(elems, x -> exists(x.b, x -> x.a == x)) AS goodElems …
                                 ^^^^^^^^^^^^^
{code}
and is obviously wrong. In given example, it will produce {_}AnalysisException{_}, however it can also silently execute wrong code (i.e., imagine there actually is dataframe column _x_ or something).

My current workaround is a reflection hack to call {{{}functions.withExpr{}}}, but it's a really bad one.

What should probably be done is instead of hard-coded name {{x}} there should be a generated unique variable name, or even some proper locally bound resolved variables (because at the moment of lambda creation this variable can be considered already resolved), however there are concerns about how that name would be displayed to an end user if there is an analysis error. Sorry, but at the moment of reporting this issue I have no ideas how to solve that.
Bug: Differentiate scalar and table function lookup in LookupFunctions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/22 22:29
Affects Version/s: 3.3.0

Currently, the LookupFunctions rule looks up unresolved scalar functions using both the scalar function registry and the table function registry. We should differentiate scalar and table function lookup in the Analyzer rule LookupFunctions.
Bug: ConstantColumnVector: a column vector with same values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/22 11:25
Affects Version/s: 3.2.0

Introduce a new column vector named `ConstantColumnVector`, it represents a column vector where every row has the same constant value.

It could help improve performance on hidden file metadata columnar file format, since metadata fields for every row in each file are exactly the same, we don't need to copy and keep multiple copies of data.
Bug: Sorting in Executors summary table in Stages Page is broken
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/Jan/22 19:48
Affects Version/s: 3.1.2

Sorting of certain columns in the Executors Summary table in the Stages Page is broken as it ignores the size units in the field value.

For example, shown in the attachment, sorting the Input Size / Records column in a decreasing order shows the error.
Bug: add support for INFORMATION_SCHEMA or other catalog variant
Status: Open | Priority: Major | Resolution: 
Created: 25/Jan/22 16:33
Affects Version/s: 3.2.0

The ability to query the metadata (from SQL) can be extremely useful.  There are ways to get at the metadata via python/scala/whatever but not from within SQL.

Given that this is a widely adapted part of SQL92, it seems like a major omission not to be supported in Spark.
Bug: Push-based shuffle disabled due to insufficient mergeLocations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/22 02:52
Affects Version/s: 3.1.0

The current shuffle merger locations is obtained based on the host of the active or dead Executors.
When dynamic executor allocation is enabled, when an application submits the first few stages, there are often not enough locations to satisfy the push merge, which causes most shuffles to not benefit from the push bashed shuffle.
The first few shuffle write stages of spark applications are generally the stages for reading tables or data sources, which account for a large amount of shuffled data. Because push merge shuffle is disabled, the end-to-end improvement of spark applications is very limited.

I probably thought of a way, but not sure if it's possible：
 *  Lazy initialize shuffle merger locations, After the mapper writes the local shuffle data, it obtains the merge location in the push thread.

Looking for advice and solutions on this issue
Bug: Deletes columns with all Null as default.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Jan/22 23:05
Affects Version/s: 3.2.0

Spark 3.2.1-RC2 
During write.json spark deletes columns with all Null as default. 
 

Spark does have dropFieldIfAllNull	false as default, according to https://spark.apache.org/docs/latest/sql-data-sources-json.html

{code:java}



from pyspark import pandas as ps
import re
import numpy as np
import os
import pandas as pd

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.functions import concat, concat_ws, lit, col, trim, expr
from pyspark.sql.types import StructType, StructField, StringType,IntegerType

os.environ["PYARROW_IGNORE_TIMEZONE"]="1"

def get_spark_session(app_name: str, conf: SparkConf):
    conf.setMaster('local[*]')
    conf \
      .set('spark.driver.memory', '64g')\
      .set("fs.s3a.access.key", "minio") \
      .set("fs.s3a.secret.key", "") \
      .set("fs.s3a.endpoint", "http://192.168.1.127:9000") \
      .set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
      .set("spark.hadoop.fs.s3a.path.style.access", "true") \
      .set("spark.sql.repl.eagerEval.enabled", "True") \
      .set("spark.sql.adaptive.enabled", "True") \
      .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
      .set("spark.sql.repl.eagerEval.maxNumRows", "10000") \
      .set("sc.setLogLevel", "error")
   
    return SparkSession.builder.appName(app_name).config(conf=conf).getOrCreate()

spark = get_spark_session("Falk", SparkConf())

d3 = spark.read.option("multiline","true").json("/home/jovyan/notebooks/falk/data/norm_test/3/*.json")

import pyspark
def sparkShape(dataFrame):
    return (dataFrame.count(), len(dataFrame.columns))
pyspark.sql.dataframe.DataFrame.shape = sparkShape
print(d3.shape())


(653610, 267)


d3.write.json("d3.json")


d3 = spark.read.json("d3.json/*.json")

import pyspark
def sparkShape(dataFrame):
    return (dataFrame.count(), len(dataFrame.columns))
pyspark.sql.dataframe.DataFrame.shape = sparkShape
print(d3.shape())

(653610, 186)
{code}
Bug: dev/reformat version required warning is confused
Status: In Progress | Priority: Major | Resolution: 
Created: 23/Jan/22 02:16
Affects Version/s: 3.3.0

Oh no! 💥 💔 💥 The required version `21.12b0` does not match the running version `21.7b0`!

But I'm don't it's the warnning is for `black` before I take a look on dev/reformat code.
Bug: Take EXTERNAL as a reserved table property
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/22 04:19
Affects Version/s: 3.3.0

At now. the {{EXTERNAL}} is not table reserved property. we should make {{EXTERNAL}} a truly reserved property [discuss|https://github.com/apache/spark/pull/35204#issuecomment-1014752053]
Bug: Filter with subexpression elimination may cause query failed
Status: Open | Priority: Major | Resolution: 
Created: 13/Jan/22 12:10
Affects Version/s: 3.2.0

 

The following test results will fail, the root cause was that the execution order of filter predicates had changed after subexpression elimination. So I think we should keep predicates execution order after subexpression elimination.
{code:java}
test("filter with subexpression elimination may cause query failed.") {
  withSQLConf((SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key, "false")) {
    val df = Seq(-1, 1, 2).toDF("c1")

    //register `plusOne` udf, and the function will failed if input was not a positive number.
    spark.sqlContext.udf.register("plusOne",
      (n: Int) => { if (n >= 0) n + 1 else throw new SparkException("Must be positive number.") })

    val result = df.filter("c1 >= 0 and plusOne(c1) > 1 and plusOne(c1) < 3").collect()
    assert(result.size === 1)
  }
} 

Caused by: org.apache.spark.SparkException: Must be positive number.
    at org.apache.spark.sql.DataFrameSuite.$anonfun$new$3(DataFrameSuite.scala:67)
    at scala.runtime.java8.JFunction1$mcII$sp.apply(JFunction1$mcII$sp.java:23)
    ... 20 more{code}
 

https://github.com/apache/spark/blob/0e186e8a19926f91810f3eaf174611b71e598de6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratePredicate.scala#L63

!image-2022-01-13-20-22-09-055.png!
Bug: Implement vectorized BYTE_STREAM_SPLIT encoding for Parquet V2 support
Status: Open | Priority: Major | Resolution: 
Created: 20/Jan/22 19:34
Affects Version/s: 3.2.0

Parquet V2 has a BYTE_STREAM_SPLIT encoding which is not currently directly usable by Spark (as there is no way to enable writing of this encoding). However, other engines may write a file with this encoding and the vectorized reader should be able to consume it.
A vectorized version of this encoding should be implemented
Bug: Directly call super.getDefaultReadLimit when scala issue 12523 is fixed
Status: Open | Priority: Major | Resolution: 
Created: 20/Jan/22 19:19
Affects Version/s: 3.2.0

In regards to [https://github.com/apache/spark/pull/35238] and more specifically these lines:

 

[https://github.com/apache/spark/pull/35238/files#diff-c9def1b07e12775929ebc58e107291495a48461f516db75acc4af4b6ce8b4dc7R106]

 

[https://github.com/apache/spark/pull/35238/files#diff-1ab03f750a3cbd95b7a84bb0c6bb6c6600c79dcb20ea2b43dd825d9aedab9656R140]

 

 
{code:java}
maxOffsetsPerTrigger.map(ReadLimit.maxRows).getOrElse(super.getDefaultReadLimit) {code}
 

 

needed to be changed to

 
{code:java}
maxOffsetsPerTrigger.map(ReadLimit.maxRows).getOrElse(ReadLimit.allAvailable()) {code}
 

Because of a bug in the scala compiler documented here:

 

[https://github.com/scala/bug/issues/12523]

 

After this bug is fixed we can revert this change, i.e. back to using `super.getDefaultReadLimit`
Bug: Spark executor self-exiting due to driver disassociated in Kubernetes with client deploy-mode
Status: Closed | Priority: Major | Resolution: Invalid
Created: 14/Jan/22 12:51
Affects Version/s: 3.2.0

I have Spark driver running in a Kubernetes pod with client deploy-mode and it tries to start an executor.
Executor will fail with error:

    \{"type":"log", "level":"ERROR", "name":"STREAMING_OTHERS", "time":"2022-01-14T12:29:38.318Z", "timezone":"UTC", "class":"dispatcher-Executor", "method":"spark.executor.CoarseGrainedExecutorBackend.logError(73)", "log":"Executor self-exiting due to : Driver 192-168-39-71.mni-system.pod.cluster.local:40752 disassociated! Shutting down.\n"}

Then driver will attempt to start another executor which fails with same error and this goes on and on.

In the driver pod, I see only following errors:

    22/01/14 12:26:32 ERROR TaskSchedulerImpl: Lost executor 1 on 192.168.43.250:
    22/01/14 12:27:16 ERROR TaskSchedulerImpl: Lost executor 2 on 192.168.43.233:
    22/01/14 12:27:59 ERROR TaskSchedulerImpl: Lost executor 3 on 192.168.43.221:
    22/01/14 12:28:43 ERROR TaskSchedulerImpl: Lost executor 4 on 192.168.43.217:
    22/01/14 12:29:27 ERROR TaskSchedulerImpl: Lost executor 5 on 192.168.43.197:
    22/01/14 12:30:10 ERROR TaskSchedulerImpl: Lost executor 6 on 192.168.43.237:
    22/01/14 12:30:53 ERROR TaskSchedulerImpl: Lost executor 7 on 192.168.43.196:
    22/01/14 12:31:42 ERROR TaskSchedulerImpl: Lost executor 8 on 192.168.43.228:
    22/01/14 12:32:31 ERROR TaskSchedulerImpl: Lost executor 9 on 192.168.43.254:
    22/01/14 12:33:14 ERROR TaskSchedulerImpl: Lost executor 10 on 192.168.43.204:
    22/01/14 12:33:57 ERROR TaskSchedulerImpl: Lost executor 11 on 192.168.43.231:

What is wrong? And how can I get executors running correctly?
Bug: Cannot fetch remote jar correctly
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/22 09:18
Affects Version/s: 3.1.2

When my Jar link address is encoded, the Jar cannot be pulled correctly

Log:

!image-2022-01-19-17-18-24-795.png!

 

My static file server(tomcat) log:

!image-2022-01-19-17-21-53-011.png!
Bug: Improve Rebalance statistics estimation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/22 03:47
Affects Version/s: 3.3.0

The defualt statistics estimation only consider the size in bytes, which may lost the row rount and columns statistics.

The `RebalancePartitions` actually does not change the statistics of plan, so we can use the statistics of its child for more accurate.
Bug: Improve RebalancePartitions in rules of Optimizer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/22 02:43
Affects Version/s: 3.3.0

After SPARK-37267, we support do optimize rebalance partitions in everywhere of plan rather than limit to the root node. So It should make sense to also let `RebalancePartitions` work in all rules of Optimizer like `Repartition` and `RepartitionByExpression` did.
Bug: Support ALTER TABLE SWAP - atomically swap all content and metadata between two tables
Status: Open | Priority: Major | Resolution: 
Created: 17/Jan/22 06:19
Affects Version/s: 3.3.0

Add ALTER TABLE SWAP to improve user experience with Spark SQL. Other DBMSs have the feature:
 * Snowflake: [https://docs.snowflake.com/en/sql-reference/sql/alter-table.html#parameters]
 * MySQL(not exactly the same statement but the same transaction can be done): [https://dev.mysql.com/doc/refman/8.0/en/rename-table.html]
Bug: Upgrade Scala to 2.13.8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/22 14:06
Affects Version/s: 3.3.0

Scala 2.13.8 already tags:

[https://github.com/scala/scala/releases/tag/v2.13.8]

 

https://contributors.scala-lang.org/t/scala-2-13-8-release-planning/5487
Bug: SQL tables created with JDBC with Spark 3.1 are not readable with 3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/22 22:47
Affects Version/s: 3.2.0

In https://github.com/apache/spark/blob/bd24b4884b804fc85a083f82b864823851d5980c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L312, a new metadata field is added during reading. As we do a full comparison of the user-provided schema and the actual schema in https://github.com/apache/spark/blob/bd24b4884b804fc85a083f82b864823851d5980c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L356, resolution fails if a table created with Spark 3.1 is read with Spark 3.2.
Bug: Support Parquet v2 data page RLE encoding (for Boolean Values) for the vectorized path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/22 08:24
Affects Version/s: 3.3.0

Parquet v2 data page write Boolean Values use RLE encoding, when read v2 boolean type values it will throw exceptions as follows now:

 
{code:java}
Caused by: java.lang.UnsupportedOperationException: Unsupported encoding: RLE
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.getValuesReader(VectorizedColumnReader.java:305) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.initDataReader(VectorizedColumnReader.java:277) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV2(VectorizedColumnReader.java:344) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.access$100(VectorizedColumnReader.java:48) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:250) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader$1.visit(VectorizedColumnReader.java:237) ~[classes/:?]
    at org.apache.parquet.column.page.DataPageV2.accept(DataPageV2.java:192) ~[parquet-column-1.12.2.jar:1.12.2]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPage(VectorizedColumnReader.java:237) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:173) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:311) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:209) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:116) ~[classes/:?]
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:298) ~[classes/:?]
    ... 19 more {code}
Bug: IllegalStateException when transforming an array inside a nested struct
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/22 14:49
Affects Version/s: 3.2.0

*NOTE*: this bug is only present in version {{3.2.0}}. Downgrading to {{3.1.2}} solves the problem.

h3. Prerequisites to reproduce the bug

# use Spark version 3.2.0
# create a DataFrame with an array field, which contains a struct field with a nested array field
# *apply a limit* to the DataFrame
# transform the outer array, renaming one of its fields
# transform the inner array too, which requires two {{getField}} in sequence

h3. Example that reproduces the bug

This is a minimal example (as minimal as I could make it) to reproduce the bug:

{code}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.{DataFrame, Row}

def makeInput(): DataFrame = {
    val innerElement1 = Row(3, 3.12)
    val innerElement2 = Row(4, 2.1)
    val innerElement3 = Row(1, 985.2)
    val innerElement4 = Row(10, 757548.0)
    val innerElement5 = Row(1223, 0.665)

    val outerElement1 = Row(1, Row(List(innerElement1, innerElement2)))
    val outerElement2 = Row(2, Row(List(innerElement3)))
    val outerElement3 = Row(3, Row(List(innerElement4, innerElement5)))

    val data = Seq(
        Row("row1", List(outerElement1)),
        Row("row2", List(outerElement2, outerElement3)),
    )

    val schema = new StructType()
        .add("name", StringType)
        .add("outer_array", ArrayType(new StructType()
            .add("id", IntegerType)
            .add("inner_array_struct", new StructType()
                .add("inner_array", ArrayType(new StructType()
                    .add("id", IntegerType)
                    .add("value", DoubleType)
                ))
            )
        ))

    spark.createDataFrame(spark.sparkContext
        .parallelize(data),schema)
}

// val df = makeInput()
val df = makeInput().limit(2)
// val df = makeInput().limit(2).cache()

val res = df.withColumn("extracted", transform(
    col("outer_array"),
    c1 => {
        struct(
            c1.getField("id").alias("outer_id"),
            transform(
                c1.getField("inner_array_struct").getField("inner_array"),
                c2 => {
                    struct(
                        c2.getField("value").alias("inner_value")
                    )
                }
            )
        )
    }
))

res.printSchema()
res.show(false)
{code}

h4. Executing the example code

When executing it as-is, the execution will fail on the {{show}} statement, with

{code}
java.lang.IllegalStateException Couldn't find _extract_inner_array#23 in [name#2,outer_array#3]
{code}

However, *if the limit is not applied, or if the DataFrame is cached after the limit, everything works* (you can uncomment the corresponding lines in the example to try it).
Bug: Put the some built-in table properties into V1Table.propertie to adapt to V2 command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 09:00
Affects Version/s: 3.3.0

At now, we have no built-in table properties in V1Table.propertie, So we could not get the correct result for some V2 command to run for V1Table.

eg: `SHOW CREATE TABLE`(V2), we get the provider,location,comment,options from the table.properties. but now we have nothing in table.proerties.