av9ash/GitBugs huggingface dataset:
Bug: equality comparison for map type field
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 06/Dec/21 14:20
Affects Version/s: 2.4.0

hi

where I try to execute this kind of qury 

 
{code:java}
SELECT * FROM myTable where myMapField is null {code}
I get this king of error 

 
{code:java}
org.apache.spark.sql.AnalysisException: Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column is map ... {code}
According to this link [https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L406] 

 

would  it be possible to assign a default comparaison function for Map type  ?

i'm working on spark 2.4, and it seams that this issue happens with version 3.2 too (same comment in 3.2 branch)
Bug: V2 ReplaceTableAsSelect command should qualify location
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/21 23:10
Affects Version/s: 3.3.0

V2 ReplaceTableAsSelect command should qualify location. Currently, 


{code:java}
spark.sql("REPLACE TABLE testcat.t USING foo LOCATION '/tmp/foo' AS SELECT id FROM source")
spark.sql("DESCRIBE EXTENDED testcat.t").show(false)
{code}

displays the location as `/tmp/foo` whereas V1 command displays/stores it as qualified (`file:/tmp/foo`).
Bug: Support set parallel through data source properties
Status: In Progress | Priority: Major | Resolution: 
Created: 05/Dec/21 12:15
Affects Version/s: 3.3.0

{code:scala}
spark.read.option("Parallel", 10).parquet("path/to/parquet")
{code}

{code:sql}
CREATE TABLE test_parallel (
  foo STRING,
  bar STRING)
USING parquet
OPTIONS (
  PARALLEL '1000'
)
{code}
Bug: V2 CreateTableAsSelect command should qualify location
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/21 06:51
Affects Version/s: 3.3.0

V2 CreateTableAsSelect command should qualify location. Currently, 

 
{code:java}
spark.sql("CREATE TABLE testcat.t USING foo LOCATION '/tmp/foo' AS SELECT id FROM source")
spark.sql("DESCRIBE EXTENDED testcat.t").show(false)
{code}
displays the location as `/tmp/foo` whereas V1 command displays/stores it as qualified (`[file:/tmp/foo|file:///tmp/foo]`).
Bug: Optimize the dynamic partitioning prune rules to avoid inserting unnecessary predicates to improve performance
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Dec/21 05:46
Affects Version/s: 3.0.0

Currently, the dynamic partition pruning rule will insert a predicate on the filterable table using the filter from the other side of the join and a custom wrapper called DynamicPruning，and the predicate will be re-optimized by the AQE or non-AQE.

But, sometimes the predicate may be unnecessary if the join can NOT reuse broadcastExchange or it is not benefit，and it will be dropped by the rules of  the AQE or non-AQE.

We should optimize the dynamic partitioning pruning rule to avoid inserting unnecessary predicates to improve performance.
Bug: Support Relation With LateralView
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Dec/21 10:32
Affects Version/s: 3.1.2

{code:java}
CREATE TABLE person (id INT, name STRING, age INT, class INT, address STRING);
INSERT INTO person VALUES
    (100, 'John', 30, 1, 'Street 1'),
    (200, 'Mary', NULL, 1, 'Street 2'),
    (300, 'Mike', 80, 3, 'Street 3'),
    (400, 'Dan', 50, 4, 'Street 4');
SELECT *
FROM person AS P1
LATERAL VIEW EXPLODE(ARRAY(30, 60)) CC1 AS C_AGE1
LEFT JOIN person P2
LATERAL VIEW EXPLODE(ARRAY(50)) CC2 AS C_AGE2 ON  P1.ID = P2.ID  AND CC1.C_AGE1=P2.AGE;
{code}
{code:java}
Error msg:
LEFT JOIN person P2 ^^^ LATERAL VIEW EXPLODE(ARRAY(50)) CC2 AS C_AGE2 ON  P1.ID = P2.ID  AND CC1.C_AGE1=P2.AGE  
      .........
        at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.spark.sql.catalyst.parser.ParseException:  mismatched input 'LEFT' expecting {<EOF>, ';'}(line 4, pos 0) {code}
Bug: date +/- interval with only day-time fields returns different data type between Spark3.2 and Spark3.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/21 06:32
Affects Version/s: 3.3.0


{code:java}
select date '2011-11-11' + interval 12 hours;
{code}
 Previously returned the date type, now it returns the timestamp type.
Bug: Expose percentile function in Scala/Python APIs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Dec/21 20:13
Affects Version/s: 3.2.0

SQL offers a percentile function (exact, not approximate) that is not available directly in the Scala or Python DataFrame APIs.

While it is possible to invoke SQL functions from Scala or Python via {{{}expr(){}}}, I think most users expect function parity across Scala, Python, and SQL.
Bug: Make ColumnarToRowExec plan canonicalizable after (de)serialization
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Dec/21 10:15
Affects Version/s: 3.0.3

SPARK-23731 fixed the plans to be serializable by leveraging lazy but SPARK-28213 introduced new code path that calls the lazy val which triggers null point exception in https://github.com/apache/spark/blob/77b164aac9764049a4820064421ef82ec0bc14fb/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala#L68

This can fail during canonicalization during, for example, eliminating sub common expressions:

{code}
java.lang.NullPointerException
    at org.apache.spark.sql.execution.FileSourceScanExec.supportsColumnar$lzycompute(DataSourceScanExec.scala:280)
    at org.apache.spark.sql.execution.FileSourceScanExec.supportsColumnar(DataSourceScanExec.scala:279)
    at org.apache.spark.sql.execution.InputAdapter.supportsColumnar(WholeStageCodegenExec.scala:509)
    at org.apache.spark.sql.execution.ColumnarToRowExec.<init>(Columnar.scala:67)
    ...
    at org.apache.spark.sql.catalyst.plans.QueryPlan.canonicalized$lzycompute(QueryPlan.scala:581)
    at org.apache.spark.sql.catalyst.plans.QueryPlan.canonicalized(QueryPlan.scala:580)
    at org.apache.spark.sql.execution.ScalarSubquery.canonicalized$lzycompute(subquery.scala:110)
    ...
    at org.apache.spark.sql.catalyst.expressions.ExpressionEquals.hashCode(EquivalentExpressions.scala:275)
    ...
    at scala.collection.mutable.HashTable.findEntry$(HashTable.scala:135)
    at scala.collection.mutable.HashMap.findEntry(HashMap.scala:44)
    at scala.collection.mutable.HashMap.get(HashMap.scala:74)
    at org.apache.spark.sql.catalyst.expressions.EquivalentExpressions.addExpr(EquivalentExpressions.scala:46)
    at org.apache.spark.sql.catalyst.expressions.EquivalentExpressions.addExprTreeHelper$1(EquivalentExpressions.scala:147)
    at org.apache.spark.sql.catalyst.expressions.EquivalentExpressions.addExprTree(EquivalentExpressions.scala:170)
    at org.apache.spark.sql.catalyst.expressions.SubExprEvaluationRuntime.$anonfun$proxyExpressions$1(SubExprEvaluationRuntime.scala:89)
    at org.apache.spark.sql.catalyst.expressions.SubExprEvaluationRuntime.$anonfun$proxyExpressions$1$adapted(SubExprEvaluationRuntime.scala:89)
    at scala.collection.immutable.List.foreach(List.scala:392)
{code}
Bug: Add Spark History Server Links for Kubernetes & other CMs
Status: Open | Priority: Minor | Resolution: 
Created: 06/Dec/21 19:00
Affects Version/s: 3.2.1

In YARN we have the Spark history server configured with `spark.yarn.historyServer.address` which allows us to print out useful links on startup for eventual debugging. More than just YARN can have the history server. We should either add `spark.kubernetes.historyServer.address` or move it to `spark.historyServer.address` w/a fall back to the old YARN specific config.
Bug: PySpark date_add only accepts an integer as it's second parameter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Dec/21 22:21
Affects Version/s: 3.2.0

Hello,

I have a quick question regarding the PySpark date_add function (and it's related functions I guess). Using date_add as an example, the PySpark API takes a [column, and an int as it's second parameter.|#L2203]]

This feels a bit weird, since the underlying SQL expression can take a column as the second parameter also- in fact, to my limited understanding, the scala [API itself|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L3114] just calls lit on this second parameter anyway. Is there a reason date_add doesn't support a column type as the second parameter in PySpark?

This isn't a major issue, as the alternative is of course to just use date_add in an expr statement- I just wondered what the usability is being traded off for. I'm happy to contribute a PR if this is something that would be worthwhile pursuing.
Bug: Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Dec/21 06:34
Affects Version/s: 3.3.0

{code}
>>> ps.range(10)
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the static sql configurations will not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some spark core configurations may not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the static sql configurations will not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some spark core configurations may not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the static sql configurations will not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some spark core configurations may not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the static sql configurations will not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some spark core configurations may not take effect.
...
   id
0   0
1   1
2   2
3   3
4   4
5   5
6   6
7   7
8   8
9   9
{code}
Bug: Suppress warnings for live pandas-on-Spark quickstart notebooks
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Dec/21 05:06
Affects Version/s: 3.2.0

https://mybinder.org/v2/gh/apache/spark/9e614e265f?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_ps.ipynb shows a bunch of warnings. We should better hide them as it's a quick start guide for users.
Bug:  Argument 1 to "rename" of "DataFrame" has incompatible type in pandas.LocIndexerLike.__getitem__
Status: Open | Priority: Minor | Resolution: 
Created: 05/Dec/21 15:07
Affects Version/s: 3.3.0

When I run dev mypy ({{mypy @ git+git://github.com/python/mypy.git@f1eb04ad379fa7492b88794d2c575461079b478a}}) against [{{feba5ac32f2598f6ca8a274850934106be0db64d}}|https://github.com/apache/spark/commit/feba5ac32f2598f6ca8a274850934106be0db64d] I get 

{code:python}
python/pyspark/pandas/indexing.py:558: error: Argument 1 to "rename" of "DataFrame" has incompatible type "Union[Any, Tuple[Any, ...]]"; expected "Union[Dict[Any, Any], Callable[[Any], Any]]"  [arg-type]
{code}

I didn't investigate if further yet, so it is not clear if it is a problem with annotations or bug in mypy.
Bug: from_json documentation lacks examples for complex types
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Dec/21 12:26
Affects Version/s: 3.2.0

The documentation for from_json currently contains only examples of getting struct value from jsonStr with simple schema. It can take a lot of time to figure out how to parse jsonStr with complex types, especially for beginners.

As it is a commonly used function, I suggest adding an additional example to demonstrate how to parse jsonStr with nested types including ArrayType and StructType, which can make the documentation more intuitive.
Bug: Simplify fetchBlocks w/o retry
Status: In Progress | Priority: Minor | Resolution: 
Created: 08/Dec/21 04:34
Affects Version/s: 3.3.0

 
{code:java}
if (maxRetries > 0) {
  // Note this Fetcher will correctly handle maxRetries == 0; we avoid it just in case there's
  // a bug in this code. We should remove the if statement once we're sure of the stability.
  new RetryingBlockTransferor(transportConf, blockFetchStarter, blockIds, listener).start()
} else {
  blockFetchStarter.createAndStart(blockIds, listener)
}{code}
Bug: Optimizer exceeds max iterations
Status: Open | Priority: Minor | Resolution: 
Created: 20/Dec/21 12:52
Affects Version/s: 3.2.0

A specific scenario causing Spark's failure in tests and a warning in production:

21/12/20 06:45:24 WARN BaseSessionStateBuilder$$anon$2: Max iterations (100) reached for batch Operator Optimization before Inferring Filters, please set 'spark.sql.optimizer.maxIterations' to a larger value.
21/12/20 06:45:24 WARN BaseSessionStateBuilder$$anon$2: Max iterations (100) reached for batch Operator Optimization after Inferring Filters, please set 'spark.sql.optimizer.maxIterations' to a larger value.

 

To reproduce run the following commands in `spark-shell`:

{{// define case class for a struct type in an array}}

{{case class S(v: Int, v2: Int)}}

 

{{// prepare a table with an array of structs}}

{{Seq((10, Seq(S(1, 2)))).toDF("i", "data").write.saveAsTable("tbl")}}

 

{{// select using SQL and join with a dataset using "left_anti"}}

{{spark.sql("select i, data[size(data) - 1].v from tbl").join(Seq(10).toDF("i"), Seq("i"), "left_anti").show()}}

 

The following conditions are required:
 # Having additional `v2` field in `S`
 # Using `{{{}data[size(data) - 1]{}}}` instead of `{{{}element_at(data, -1){}}}`
 # Using `{{{}left_anti{}}}` in join operation

 

The same behavior was observed in `master` branch and `3.1.1`.
Bug: Reduce default page size by LONG_ARRAY_OFFSET if G1GC and ON_HEAP are used
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Dec/21 09:18
Affects Version/s: 3.3.0

Spark's tungsten memory model usually tries to allocate memory by one `page` each time and allocated by long[pageSizeBytes/8] in HeapMemoryAllocator.allocate. 

Remember that java long array needs extra object header (usually 16 bytes in 64bit system), so the really bytes allocated is pageSize+16.

Assume that the G1HeapRegionSize is 4M and pageSizeBytes is 4M as well. Since every time we need to allocate 4M+16byte memory, so two regions are used with one region only occupies 16byte. Then there are about 50% memory waste.
It can happenes under different combinations of G1HeapRegionSize (varies from 1M to 32M) and pageSizeBytes (varies from 1M to 64M).
Bug: Support optimize skewed join through union
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Dec/21 12:20
Affects Version/s: 3.2.0

`OptimizeSkewedJoin` rule will take effect only when the plan has two ShuffleQueryStageExec。

With `Union`, it might break the assumption. For example, the following plans
*scenes 1*
{noformat}
Union
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage
{noformat}
*scenes 2*
{noformat}
Union
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage
    HashAggregate
{noformat}
when one or more of the SMJ data in the above plan is skewed, it cannot be processed at present.

It's better to support partial optimize with Union.
Bug: ColumnOrName vs Column in PySpark Functions module
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Dec/21 14:32
Affects Version/s: 3.2.0

PySpark has mainly migrated to supporting both Column input types as well as string names of columns ("ColumnOrName") in it's functions module. There seem to be a small number of functions that need updating; either on conversions of input string names representing columns into the Column type, or simple annotation changes that indicate the function supports column string names.

Below are the functions I've seen:
 * F.overlay: Annotation only
 * F.least: Annotation only
 * F.slice: Needs a conversion
 * F.array_repeat: Needs a conversion

See here for additional context: [https://github.com/apache/spark/pull/35032#issuecomment-1003033776]

I'm happy to make a quick PR fixing these, if there is no reason for these functions being handled as a special case.
Bug: log4j2-defaults.properties is not working since log4j 2 is always initialized by default
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Dec/21 16:47
Affects Version/s: 3.3.0

the code used to check initialization is shown below.

      val log4j2Initialized = !LogManager.getRootLogger
        .asInstanceOf[org.apache.logging.log4j.core.Logger].getAppenders.isEmpty

That works for log4j. However, log4j2 provides a default configuration so there is always a appender (ConsoleAppender) with error level.

 

reference from [https://logging.apache.org/log4j/2.x/manual/configuration.html#AutomaticConfiguration]

Log4j will provide a default configuration if it cannot locate a configuration file. The default configuration, provided in the DefaultConfiguration class, will set up:
 * A [ConsoleAppender|https://logging.apache.org/log4j/2.x/log4j-core/apidocs/org/apache/logging/log4j/core/appender/ConsoleAppender.html] attached to the root logger.
 * A [PatternLayout|https://logging.apache.org/log4j/2.x/log4j-core/apidocs/org/apache/logging/log4j/core/layout/PatternLayout.html] set to the pattern "%d\{HH:mm:ss.SSS} [%t] %-5level %logger\{36} - %msg%n" attached to the ConsoleAppender

Note that by default Log4j assigns the root logger to Level.ERROR.
Bug: Try to remove `--add-exports` compile option for Java 17
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Dec/21 03:10
Affects Version/s: 3.3.0

We can remove the `--add-exports` compile option for Java 17  introduced by SPARK-37070 by doing the following:
 # Introduce test dependence: mockito-inline  
 # Upgrade mockito version from 3.4.x to 3.12.x
 # Upgrade org.scalatestplus:mockito from 3-4 to 3-12
Bug: Python UDF fails when it should not get evaluated
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 27/Dec/21 09:55
Affects Version/s: 2.3.4

Haven't checked on newer versions yet.

If i define in Python:
{code:java}
def udf1(col1):
    print(col1[2])
    return "blah"

spark.udf.register("udf1", udf1) {code}
and then use it in SQL:
{code:java}
select case when length(c)>2 then udf1(c) end
from (
    select explode(array("123","234","12")) as c
) {code}
it fails on:
{noformat}
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in main
    process()
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 248, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 155, in <lambda>
    func = lambda _, it: map(mapper, it)
  File "<string>", line 1, in <lambda>
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 76, in <lambda>
    return lambda *a: f(*a)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/util.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "<stdin>", line 3, in udf1
IndexError: string index out of range{noformat}
Although in the out-of-range row it should not get evaluated at all as the case-when filters for lengths of more than 2 letters.

the same scenario works great when we define instead a Scala UDF.

will check now if it happens also for newer versions.
Bug: Demo sql is incorrect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Dec/21 03:06
Affects Version/s: 3.2.0

There is a sql statement in this section https://spark.apache.org/docs/latest/sql-ref-null-semantics.html#null-semantics that is incorrect.
{code:java}
SELECT * FROM person GROUP BY age HAVING max(age) > 18; {code}
should be
{code:java}
SELECT age, count(*) FROM person GROUP BY age HAVING max(age) > 18;{code}
Bug: Spark master UI behind reverse proxy app storage redirects to master home page
Status: Open | Priority: Minor | Resolution: 
Created: 23/Dec/21 09:43
Affects Version/s: 3.0.1

1. Using a Spark v3.1.1 Cluster with haproxy as reverse proxy enabled, web UI is at: [http://spark.internal.domain:8080/]
2. Starting an application and enter the application specific web UI (by clicking on the running application name): [http://spark.internal.domain:8080/app/?appId=app-20211202144454-0119]
3. Click on "Application Detail UI" and you will get redirected to [http://spark.internal.domain:8080|http://spark.internal.domain:8080/] [/proxy/app-20211202144454-0119/jobs/|http://sparkm-v3-mlm-op.itn.intraorange/proxy/app-20211202144454-0119/jobs/] (notice the "proxy" in the URL)
4. If you click on any link (e.g. on the "Stages" or "Storage") the Spark master home page UI will be served again, not "stages" not "storage".
5. If you insert the "proxy/app-20211202144454-0119/" and add the stages/ or storage/ in the URL it works.
Bug: Upgrade Netty from 4.1.68 to 4.1.72
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Dec/21 11:29
Affects Version/s: 3.3.0

After SPARK-36729, netty released 3 new versions:
 * [https://netty.io/news/2021/10/11/4-1-69-Final.html]
 * [https://netty.io/news/2021/10/11/4-1-70-Final.html]
 * [https://netty.io/news/2021/12/09/4-1-71-Final.html]
 * [https://netty.io/news/2021/12/13/4-1-72-Final.html]

We can avoid some potential bugs by upgrading to 4.1.72 and 4.1.72 also fixed security issues related to log4j2 in Netty
Bug: Improve logging in BroadcastExchangeExec
Status: In Progress | Priority: Minor | Resolution: 
Created: 23/Dec/21 01:17
Affects Version/s: 2.4.6

The logging and errors from inside
{code:java}

override lazy val relationFuture: Future[broadcast.Broadcast[Any]] = {
  SQLExecution.withThreadLocalCaptured[broadcast.Broadcast[Any]](
    session, BroadcastExchangeExec.executionContext) {
        try {
.....
        } {code}
are hard to relate to the right join in a plan with lost of Broadcast joins.
Spark UI displays {noformat}BroadcastExchangeExec.runId{noformat} but it's not in the logs.
Bug: Support sort aggregate code-gen without grouping keys
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Dec/21 05:13
Affects Version/s: 3.3.0

We can support `SortAggregateExec` code-gen without grouping keys. When there's no grouping key, sort aggregate should share same execution logic with hash aggregate. Refactor the code-gen without grouping keys, from `HashAggregateExec` to a base class `AggregateCodegenSupport`, and `SortAggregateExec` can share the same implementation by extending `AggregateCodegenSupport`.

The implementation of `SortAggregateExec` code-gen with grouping keys will be added in follow-up PR.
Bug: JDBC: Provide generic metadata functions
Status: Open | Priority: Minor | Resolution: 
Created: 16/Dec/21 13:09
Affects Version/s: 3.2.0

JDBC Driver vendors expose the metadata (databases/tables) for the underlying engine through the [DatabaseMetaData interface.|https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html]

Today when a user wants to fetch the metadata for an engine, they have to execute an sql statement that is tailored to a specific engine/syntax instead of using a more generic approach.

 I suggest we add two new functions to the JDBC reader: 
{code:java}
listDatabases & listTables{code}
Bug: SparkSQLCLIDriver will use hive defaults to resolve warehouse dir
Status: In Progress | Priority: Minor | Resolution: 
Created: 16/Dec/21 08:52
Affects Version/s: 3.2.0

{code:log}
21/12/16 15:27:26.713 main INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir.
21/12/16 15:27:26.761 main INFO SharedState: Warehouse path is 'file:/user/hive/warehouse'.

...
...

21/12/16 15:27:36.559 main INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
21/12/16 15:27:36.561 main INFO SharedState: Warehouse path is 'file:/Users/kentyao/Downloads/spark/spark-3.2.0-bin-hadoop3.2/spark-warehouse'.

 {code}