av9ash/GitBugs huggingface dataset:
Bug: ALTER TABLE .. DROP PARTITION uncaches Hive table while updating table stats
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/21 10:23
Affects Version/s: 3.2.0

The example below portraits the issue:
{code:scala}
scala> spark.conf.set("spark.sql.statistics.size.autoUpdate.enabled", true)

scala> sql(s"CREATE TABLE tbl (id int, part int) USING hive PARTITIONED BY (part)")
21/01/10 13:19:59 WARN HiveMetaStore: Location: file:/Users/maximgekk/proj/apache-spark/spark-warehouse/tbl specified for non-external table:tbl
res12: org.apache.spark.sql.DataFrame = []

scala> sql("INSERT INTO tbl PARTITION (part=0) SELECT 0")
res13: org.apache.spark.sql.DataFrame = []

scala> sql("INSERT INTO tbl PARTITION (part=1) SELECT 1")
res14: org.apache.spark.sql.DataFrame = []

scala> sql("CACHE TABLE tbl")
res15: org.apache.spark.sql.DataFrame = []

scala> sql("SELECT * FROM tbl").show(false)
+---+----+
|id |part|
+---+----+
|0  |0   |
|1  |1   |
+---+----+


scala> spark.catalog.isCached("tbl")
res17: Boolean = true

scala> sql("ALTER TABLE tbl DROP PARTITION (part=0)")
res18: org.apache.spark.sql.DataFrame = []

scala> spark.catalog.isCached("tbl")
res19: Boolean = false
{code}
Bug: Add rowCount for Range operator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/21 02:18
Affects Version/s: 3.2.0

{code:scala}
spark.sql("set spark.sql.cbo.enabled=true")
spark.sql("select id from range(100)").explain("cost")
{code}

Current:
{noformat}
== Optimized Logical Plan ==
Range (0, 100, step=1, splits=None), Statistics(sizeInBytes=800.0 B)
{noformat}

Expected:
{noformat}
== Optimized Logical Plan ==
Range (0, 100, step=1, splits=None), Statistics(sizeInBytes=800.0 B, rowCount=100)
{noformat}
Bug: Some operator missing rowCount when enable CBO
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jan/21 08:34
Affects Version/s: 3.2.0

Some operator missing rowCount when enable CBO, for example:
{code:scala}
spark.range(1000).selectExpr("id as a", "id as b").write.saveAsTable("t1")
spark.sql("ANALYZE TABLE t1 COMPUTE STATISTICS FOR ALL COLUMNS")
spark.sql("set spark.sql.cbo.enabled=true")
spark.sql("set spark.sql.cbo.planStats.enabled=true")
spark.sql("select * from (select * from t1 distribute by a limit 100) distribute by b").explain("cost")
{code}

Current:
{noformat}
== Optimized Logical Plan ==
RepartitionByExpression [b#2129L], Statistics(sizeInBytes=2.3 KiB)
+- GlobalLimit 100, Statistics(sizeInBytes=2.3 KiB, rowCount=100)
   +- LocalLimit 100, Statistics(sizeInBytes=23.4 KiB)
      +- RepartitionByExpression [a#2128L], Statistics(sizeInBytes=23.4 KiB)
         +- Relation[a#2128L,b#2129L] parquet, Statistics(sizeInBytes=23.4 KiB, rowCount=1.00E+3)
{noformat}

Expected:
{noformat}
== Optimized Logical Plan ==
RepartitionByExpression [b#2129L], Statistics(sizeInBytes=2.3 KiB, rowCount=100)
+- GlobalLimit 100, Statistics(sizeInBytes=2.3 KiB, rowCount=100)
   +- LocalLimit 100, Statistics(sizeInBytes=23.4 KiB, rowCount=1.00E+3)
      +- RepartitionByExpression [a#2128L], Statistics(sizeInBytes=23.4 KiB, rowCount=1.00E+3)
         +- Relation[a#2128L,b#2129L] parquet, Statistics(sizeInBytes=23.4 KiB, rowCount=1.00E+3)
{noformat}
Bug: Killing executors excluded on failure, results in additional executors being marked as excluded due to fetch failures
Status: Open | Priority: Major | Resolution: 
Created: 14/Jan/21 04:19
Affects Version/s: 3.0.0

Configuration:

 
{code:java}
spark.excludeOnFailure.enabled: true # aka deprecated spark.blacklist.enabled
spark.excludeOnFailure.application.fetchFailure.enabled: true # aka deprecated spark.blacklist.application.fetchFailure.enabled
spark.excludeOnFailure.killExcludedExecutors: true # aka deprecated spark.blacklist.killBlacklistedExecutors
{code}
 

 

 

In this case, we have noticed when a few executors are excluded due to task failures (maybe due to host issues), then those executors are killed after being excluded.

However, when other executors try to fetch shuffle blocks from these killed executors, then  these other executors also end up getting excluded due to `spark.excludeOnFailure.application.fetchFailure.enabled`.

Instead, the fetch failures in case of fetch from these excluded executors should not be considered when excluding executors based on `spark.excludeOnFailure.application.fetchFailure.enabled`
Bug: RaiseError generates too much code and may fails codegen in length check for char varchar
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/21 10:41
Affects Version/s: 3.1.0

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133928/testReport/org.apache.spark.sql.execution/LogicalPlanTagInSparkPlanSuite/q41/

We can reduce more than 8000 bytes by removing the unnecessary CONCAT expression.
Bug: SQLContext.dropTempTable fails if cache is non-empty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/21 22:05
Affects Version/s: 3.1.0

{{SQLContext.dropTempView}} calls {{CatalogImpl.dropTempView}} which is implemented as following:
{code}
  override def dropTempView(viewName: String): Boolean = {
    sparkSession.sessionState.catalog.getTempView(viewName).exists { viewDef =>
      sparkSession.sharedState.cacheManager.uncacheQuery(
        sparkSession, viewDef, cascade = false)
      sessionCatalog.dropTempView(viewName)
    }
  }
{code}

Here, the logical plan {{viewDef}} is not resolved, and when passing to {{uncacheQuery}}, it could fail at {{sameResult}} call, when canonicalized plan is compared. The error message looks like:
{code}
Invalid call to qualifier on unresolved object, tree: 'key
{code}

This can be reproduced via:
{code}
  test("XXX") {
    val t = "t"
    val v = "v"
    withTable(t) {
      withTempView(v) {
        sql(s"CREATE TABLE $t AS SELECT * FROM src")
        sql(s"CACHE TABLE $t")
        sql(s"CREATE TEMPORARY VIEW $v AS SELECT key FROM src LIMIT 10")
      }
    }
  }
{code}

through test.
Bug: ALTER TABLE .. ADD PARTITION does not update table stats
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/21 09:19
Affects Version/s: 3.0.2

The example below portraits the issue:
{code:sql}
spark-sql> create table tbl (col0 int, part int) partitioned by (part);
spark-sql> insert into tbl partition (part = 0) select 0;
spark-sql> set spark.sql.statistics.size.autoUpdate.enabled=true;
spark-sql> alter table tbl add partition (part = 1);
{code}
There are no stats:
{code:sql}
spark-sql> describe table extended tbl;
col0	int	NULL
part	int	NULL
# Partition Information
# col_name	data_type	comment
part	int	NULL

# Detailed Table Information
Database	default
Table	tbl
Owner	maximgekk
Created Time	Tue Jan 12 12:00:03 MSK 2021
Last Access	UNKNOWN
Created By	Spark 3.2.0-SNAPSHOT
Type	MANAGED
Provider	hive
Table Properties	[transient_lastDdlTime=1610442003]
Location	file:/Users/maximgekk/proj/fix-stats-in-add-partition/spark-warehouse/tbl
Serde Library	org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat	org.apache.hadoop.mapred.TextInputFormat
OutputFormat	org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Storage Properties	[serialization.format=1]
Partition Provider	Catalog
{code}

*As we can see there is no stats.* For instance, ALTER TABLE .. DROP PARTITION updates stats:
{code:sql}
spark-sql> alter table tbl drop partition (part = 1);
spark-sql> describe table extended tbl;
col0	int	NULL
part	int	NULL
# Partition Information
# col_name	data_type	comment
part	int	NULL

# Detailed Table Information
...
Statistics	2 bytes
{code}
Bug: Call updateTableStats() from AlterTableAddPartitionCommand
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 10/Jan/21 16:01
Affects Version/s: 3.2.0

AlterTableAddPartitionCommand.run does almost the same as `CommandUtils.updateTableStats` while updating table stats. The code:
{code:scala}
    if (table.stats.nonEmpty) {
      if (sparkSession.sessionState.conf.autoSizeUpdateEnabled) {
        val addedSize = CommandUtils.calculateMultipleLocationSizes(sparkSession, table.identifier,
          parts.map(_.storage.locationUri)).sum
        if (addedSize > 0) {
          val newStats = CatalogStatistics(sizeInBytes = table.stats.get.sizeInBytes + addedSize)
          catalog.alterTableStats(table.identifier, Some(newStats))
        }
      } else {
        catalog.alterTableStats(table.identifier, None)
      }
    }
{code}
can be replaced by:
{code:scala}
CommandUtils.updateTableStats()
{code}
Bug: Check stats of cached table after altering 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/21 13:04
Affects Version/s: 3.2.0

1. Port the test added by https://github.com/apache/spark/pull/31112 for `ALTER TABLE .. ADD PARTITION` and `ALTER TABLE .. RENAME PARTITION`.
2. Port the test for V1 in-memory catalog to run it for drop partitions
Bug: Shuffle batch fetch can't be disabled once it's enabled previously
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/21 13:31
Affects Version/s: 3.0.0

{code:java}
  if (SQLConf.get.fetchShuffleBlocksInBatch) {
    dependency.rdd.context.setLocalProperty(
      SortShuffleManager.FETCH_SHUFFLE_BLOCKS_IN_BATCH_ENABLED_KEY, "true")
  }
{code}
The current code has a problem that once we set `fetchShuffleBlocksInBatch` to true first, we can never disable the batch fetch even if set `fetchShuffleBlocksInBatch` to false later.
Bug: Rename all decommission configurations to use the same namespace "spark.decommission.*"
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jan/21 12:02
Affects Version/s: 3.1.0

Currently, decommission configurations are using difference namespaces, e.g., 
 * spark.decommission

 * spark.storage.decommission

 * spark.executor.decommission

which may introduce unnecessary overhead for end-users. It's better to keep them under the same namespace.
Bug: Metastore daemon threads not cleaned up with session.stop()
Status: Open | Priority: Major | Resolution: 
Created: 11/Jan/21 22:48
Affects Version/s: 2.4.3

Basically, spinning down a SparkSession using session.stop() does not appear to clean up JDBC connection pool daemon threads. Please see this StackOverflow question for more info (and a code snippet to reproduce): [https://stackoverflow.com/questions/65428297/spark-background-threads-not-cleaned-up-when-using-embedded-derby-metastore]

As mentioned in the question, this can eat up resources during unit tests where sessions are spun up/spun down throughout a test suite.

I haven't tested this against a non-embedded metastore, but this does occur when using derby in memory, too.
Bug: cipher problem with IBM JDK
Status: Open | Priority: Major | Resolution: 
Created: 11/Jan/21 14:08
Affects Version/s: 3.0.1

We get following error message in log: 

 
{noformat}
21/01/11 11:37:00 INFO Master: Running Spark version 3.0.1
21/01/11 11:37:02 WARN SslContextFactory: No supported ciphers from [SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384, SSL_RSA_WITH_AES_256_CBC_SHA256, SSL_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, SSL_ECDH_RSA_WITH_AES_256_CBC_SHA384, SSL_DHE_RSA_WITH_AES_256_CBC_SHA256, SSL_DHE_DSS_WITH_AES_256_CBC_SHA256, SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_AES_256_CBC_SHA, SSL_ECDH_ECDSA_WITH_AES_256_CBC_SHA, SSL_ECDH_RSA_WITH_AES_256_CBC_SHA, SSL_DHE_RSA_WITH_AES_256_CBC_SHA, SSL_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256, SSL_RSA_WITH_AES_128_CBC_SHA256, SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, SSL_ECDH_RSA_WITH_AES_128_CBC_SHA256, SSL_DHE_RSA_WITH_AES_128_CBC_SHA256, SSL_DHE_DSS_WITH_AES_128_CBC_SHA256, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_AES_128_CBC_SHA, SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA, SSL_ECDH_RSA_WITH_AES_128_CBC_SHA, SSL_DHE_RSA_WITH_AES_128_CBC_SHA, SSL_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, SSL_RSA_WITH_AES_256_GCM_SHA384, SSL_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, SSL_ECDH_RSA_WITH_AES_256_GCM_SHA384, SSL_DHE_DSS_WITH_AES_256_GCM_SHA384, SSL_DHE_RSA_WITH_AES_256_GCM_SHA384, SSL_ECDHE_RSA_WITH_AES_128_GCM_SHA256, SSL_RSA_WITH_AES_128_GCM_SHA256, SSL_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, SSL_ECDH_RSA_WITH_AES_128_GCM_SHA256, SSL_DHE_RSA_WITH_AES_128_GCM_SHA256, SSL_DHE_DSS_WITH_AES_128_GCM_SHA256]{noformat}
 

and can not connect to the encrypted UI port and so on. Searching for this error I found the information that [jetty blocks all ciphers starting with SSL_|https://github.com/eclipse/jetty.project/issues/3817]. Unfortunately does the IBM JDK we are using, name all ciphers with SSL_* instead of TLS_* for the newer one.

I was looking for the strings reported in the issue and found them in spark-core_2.12-3.0.1.jar, file org/sparkproject/jetty/util/ssl/SslContextFactory.class:
{code:java}
^.*_(MD5|SHA|SHA1)$
^TLS_RSA_.*$
^SSL_.*$
^.*_NULL_.*$
^.*_anon_.*${code}
I could not find the source to check if I can take influence with the configuration on these definition.

Does my analysis makes sense? And would it be possible to add a configuration to not exclude the SSL_* ciphers, just for  JDKs that define different naming schemes?
Bug: Fix empty array failed in functions.lit()
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Jan/21 13:43
Affects Version/s: 3.2.0

We will get exception if we execute code:
{code:java}
spark.range(1).select(lit(Array())){code}
error msg:
{code:java}
Exception in thread "main" org.apache.spark.sql.AnalysisException: Unsupported component type class java.lang.Object in arrays;Exception in thread "main" org.apache.spark.sql.AnalysisException: Unsupported component type class java.lang.Object in arrays; at org.apache.spark.sql.catalyst.expressions.Literal$.componentTypeToDataType(literals.scala:129) at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:80) at org.apache.spark.sql.catalyst.expressions.Literal$.$anonfun$create$2(literals.scala:148) at scala.util.Failure.getOrElse(Try.scala:222) at org.apache.spark.sql.catalyst.expressions.Literal$.create(literals.scala:148) at org.apache.spark.sql.functions$.typedLit(functions.scala:132) at org.apache.spark.sql.functions$.lit(functions.scala:115)
{code}
Bug: ALTER TABLE .. RENAME TO PARTITION doesn't refresh cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/21 10:38
Affects Version/s: 3.0.1

Here is the example to reproduce the issue:
{code:sql}
spark-sql> CREATE TABLE tbl1 (col0 int, part0 int) USING parquet PARTITIONED BY (part0);
spark-sql> INSERT INTO tbl1 PARTITION (part0=0) SELECT 0;
spark-sql> INSERT INTO tbl1 PARTITION (part0=1) SELECT 1;
spark-sql> CACHE TABLE tbl1;
spark-sql> SELECT * FROM tbl1;
0	0
1	1
spark-sql> ALTER TABLE tbl1 PARTITION (part0 = 0) RENAME TO PARTITION (part0 = 2);
spark-sql> SELECT * FROM tbl1;
0	0
1	1
{code}
Bug: UnresolvedTableOrView should retain SQL text position
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jan/21 17:52
Affects Version/s: 3.2.0

UnresolvedTableOrView should retain SQL text position. The following commands will be handled:

"DROP TABLE unknown"
"DESCRIBE TABLE unknown"
"ANALYZE TABLE unknown COMPUTE STATISTICS"
"ANALYZE TABLE unknown COMPUTE STATISTICS FOR COLUMNS col"
"ANALYZE TABLE unknown COMPUTE STATISTICS FOR ALL COLUMNS"
"SHOW CREATE TABLE unknown"
"REFRESH TABLE unknown"
"SHOW COLUMNS FROM unknown"
"SHOW COLUMNS FROM unknown IN db"

"ALTER TABLE unknown RENAME TO t"
"ALTER VIEW unknown RENAME TO v"
Bug: Add doc for "'like any' and 'like all' operators"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/21 03:59
Affects Version/s: 3.1.0

Need to update the doc for the new LIKE predicates in the following file:

[https://github.com/apache/spark/blob/master/docs/sql-ref-syntax-qry-select-like.md]
Bug: Pass json column information via pruneColumns()
Status: Open | Priority: Major | Resolution: 
Created: 05/Jan/21 19:46
Affects Version/s: 3.0.1

Currently PushDownUtils#pruneColumns only passes root fields to SupportsPushDownRequiredColumns implementation(s).
{code}
2021-01-05 19:36:07,437 (Time-limited test) [DEBUG - org.apache.spark.internal.Logging.logDebug(Logging.scala:61)] nested schema projection List(id#33, address#34, phone#36, get_json_object(phone#36, $.code) AS get_json_object(phone, $.code)#37)
2021-01-05 19:36:07,438 (Time-limited test) [DEBUG - org.apache.spark.internal.Logging.logDebug(Logging.scala:61)] nested schema StructType(StructField(id,IntegerType,false), StructField(address,StringType,true), StructField(phone,StringType,true))
{code}
The first line shows projections and the second line shows the pruned schema.

We can see that get_json_object(phone#36, $.code) is filtered. This expression retrieves field 'code' from phone json column.

We should allow json column information to be passed via pruneColumns().
Bug: Data in redis was obtained using mapPartitions Sparkstreaming, which was not as expected
Status: Open | Priority: Major | Resolution: 
Created: 05/Jan/21 12:23
Affects Version/s: 2.4.2

 
{code:java}
rdd.mapPartitions(par => {
 val jedis: Jedis = AdJedisPoolUtils.getJedis
 jedis.select(4)
 val res: Iterator[(((String, String), (String, String, String, String, String, String, String, String, String)), String)] = par.map(data =>{ 
println(data._2._1) 
val tagId: String = jedis.hget("game_tag_union", data._2._1) 
println(tagId) 
(data, tagId) 
})
jedis.close()
 res
 })
{code}
 

*The final result was completely different from the one foreachPartition got, but foreachPartition got the correct solution*

*eg:*
{code:java}
rdd.foreachPartition(par=>{
val jedis: Jedis = AdJedisPoolUtils.getJedis
 jedis.select(4)
par.foreach(data=>{ 
val tagId: String = jedis.hget("game_tag_union", data._2._1) println(tagId) 
})
jedis.close()
})
{code}
Bug: ExecutorAllocationListener threw an exception java.util.NoSuchElementException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/21 03:05
Affects Version/s: 3.0.1

21/01/04 03:00:32,259 WARN [task-result-getter-2] scheduler.TaskSetManager:69 : Lost task 306.1 in stage 600.0 (TID 283610, hdc49-mcc10-01-0510-4108-039-tess0097.stratus.rno.ebay.com, executor 27): TaskKilled (another attempt succeeded)
21/01/04 03:00:32,259 INFO [task-result-getter-2] scheduler.TaskSetManager:57 : Task 306.1 in stage 600.0 (TID 283610) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the
previous stage needs to be re-run, or because a different copy of the task has already succeeded).
21/01/04 03:00:32,259 INFO [task-result-getter-2] cluster.YarnClusterScheduler:57 : Removed TaskSet 600.0, whose tasks have all completed, from pool default
21/01/04 03:00:32,259 INFO [HiveServer2-Handler-Pool: Thread-5853] thriftserver.SparkExecuteStatementOperation:190 : Returning result set with 50 rows from offsets [5378600, 5378650) with 1fe245f8-a7f9-4ec0-bcb5-8cf324cbbb47
21/01/04 03:00:32,260 ERROR [spark-listener-group-executorManagement] scheduler.AsyncEventQueue:94 : Listener ExecutorAllocationListener threw an exception
java.util.NoSuchElementException: key not found: Stage 600 (Attempt 0)
        at scala.collection.MapLike.default(MapLike.scala:235)
        at scala.collection.MapLike.default$(MapLike.scala:234)
        at scala.collection.AbstractMap.default(Map.scala:63)
        at scala.collection.mutable.HashMap.apply(HashMap.scala:69)
        at org.apache.spark.ExecutorAllocationManager$ExecutorAllocationListener.onTaskEnd(ExecutorAllocationManager.scala:621)
        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:45)
        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:38)
        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:38)
        at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:115)
        at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:99)
        at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:116)
        at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:116)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:102)
        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:97)
        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1320)
        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:97)
Bug: Running bin/spark-sql gives NoSuchMethodError
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 05/Jan/21 00:09
Affects Version/s: 3.0.1

I ran 'mvn install -Phive -Phive-thriftserver -DskipTests'

Running bin/spark-sql gives the following error:
{code}
21/01/05 00:06:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SharedState$.loadHiveConfFile$default$3()Lscala/collection/Map;
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:136)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:934)
{code}
Scala version 2.12.10
Bug: v2 ALTER TABLE .. DROP PARTITION does not refresh cached table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/21 11:18
Affects Version/s: 3.2.0

The test below portraits the issue:
{code:scala}
  test("SPARK-33950: refresh cache after partition dropping") {
    withNamespaceAndTable("ns", "tbl") { t =>
      sql(s"CREATE TABLE $t (id int, part int) $defaultUsing PARTITIONED BY (part)")
      sql(s"INSERT INTO $t PARTITION (part=0) SELECT 0")
      sql(s"INSERT INTO $t PARTITION (part=1) SELECT 1")
      assert(!spark.catalog.isCached(t))
      sql(s"CACHE TABLE $t")
      assert(spark.catalog.isCached(t))
      QueryTest.checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(0, 0), Row(1, 1)))
      sql(s"ALTER TABLE $t DROP PARTITION (part=0)")
      assert(spark.catalog.isCached(t))
      QueryTest.checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(1, 1)))
    }
  }
{code}
The last check fails:
{code}
== Results ==
!== Correct Answer - 1 ==   == Spark Answer - 2 ==
!struct<>                   struct<id:int,part:int>
![1,1]                      [0,0]
!                           [1,1]
{code}
Bug: Add an option to enable CBO in TPCDSQueryBenchmark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/21 11:42
Affects Version/s: 3.2.0

This ticket aims at adding a new option {{--cbo}} to enable CBO in TPCDSQueryBenchmark. I think this option is useful so as to monitor performance changes with CBO enabled.