av9ash/GitBugs huggingface dataset:
Bug: [CONNECT][SCALA] Update user agent to recognize SPARK_CONNECT_USER_AGENT and include environment specific attributes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/23 15:15
Affects Version/s: 3.5.0

Similar to the [Python client,|https://github.com/apache/spark/blob/2cc1ee4d3a05a641d7a245f015ef824d8f7bae8b/python/pyspark/sql/connect/client/core.py#L284] the Scala client should:
 # Recognize SPARK_CONNECT_USER_AGENT env variable
 # Always include the OS, Java version, Scala version, and Spark version
Bug: Expand more exception constructors to support error framework parameters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/23 12:30
Affects Version/s: 4.0.0

* SparkNumberFormatException
 * SparkIllegalArgumentException
 * SparkArithmeticException
 * SparkUnsupportedOperationException
 * SparkArrayIndexOutOfBoundsException
 * SparkDateTimeException
 * SparkRuntimeException
 * SparkUpgradeException
Bug: Replace: _LEGACY_ERROR_TEMP_2196 with a better error class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/23 22:21
Affects Version/s: 4.0.0

{code:java}
def cannotFetchTablesOfDatabaseError(dbName: String, e: Exception): Throwable = {
  new SparkException(
    errorClass = "_LEGACY_ERROR_TEMP_2196",
    messageParameters = Map(
      "dbName" -> dbName),
    cause = e)
} {code}
Bug: Replace `scala.runtime.Tuple3Zipped` to `scala.collection.LazyZip3`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/23 08:38
Affects Version/s: 4.0.0


{code:java}
@scala.deprecated(message = "Use scala.collection.LazyZip3.", since = "2.13.0")
final class Tuple3Zipped[El1, It1 <: scala.Iterable[El1], El2, It2 <: scala.Iterable[El2], El3, It3 <: scala.Iterable[El3]](colls : scala.Tuple3[It1, It2, It3]) extends scala.AnyVal with scala.runtime.ZippedIterable3[El1, El2, El3] {


@scala.deprecated(message = "Use scala.collection.LazyZip3.", since = "2.13.0")
object Tuple3Zipped extends scala.AnyRef {
  final class Ops[T1, T2, T3](x : scala.Tuple3[T1, T2, T3]) extends scala.AnyVal {
    @scala.deprecated(message = "Use xs.lazyZip(yz).lazyZip(zs).map((_, _, _))", since = "2.13.0")
    def invert[El1, It1[a] <: scala.Iterable[a], El2, It2[a] <: scala.Iterable[a], El3, It3[a] <: scala.Iterable[a], That](implicit w1 : scala.<:<[T1, It1[El1]], w2 : scala.<:<[T2, It2[El2]], w3 : scala.<:<[T3, It3[El3]], bf : scala.collection.BuildFrom[T1, scala.Tuple3[El1, El2, El3], That]) : That = { /* compiled code */ }
    @scala.deprecated(message = "Use xs.lazyZip(ys).lazyZip(zs)", since = "2.13.0")
    def zipped[El1, It1 <: scala.Iterable[El1], El2, It2 <: scala.Iterable[El2], El3, It3 <: scala.Iterable[El3]](implicit w1 : scala.Function1[T1, scala.collection.IterableOps[El1, scala.Iterable, It1] with It1], w2 : scala.Function1[T2, scala.collection.IterableOps[El2, scala.Iterable, It2] with It2], w3 : scala.Function1[T3, scala.collection.IterableOps[El3, scala.Iterable, It3] with It3]) : scala.runtime.Tuple3Zipped[El1, It1, El2, It2, El3, It3] = { /* compiled code */ }
  }
{code}
Bug: Reduce memory usage of LiveStageMetrics.accumIdsToMetricType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/23 20:21
Affects Version/s: 4.0.0

This PR aims to reduce the memory consumption of {{{}LiveStageMetrics.accumIdsToMetricType{}}}, which should help to reduce driver memory usage when running complex SQL queries that contain many operators and run many jobs.

In SQLAppStatusListener, the LiveStageMetrics.accumIdsToMetricType field holds a map which is used to look up the type of accumulators in order to perform conditional processing of a stage’s metrics.

Currently, that field is derived from {{{}LiveExecutionData.metrics{}}}, which contains metrics for _all_ operators used anywhere in the query. Whenever a job is submitted, we construct a fresh map containing all metrics that have ever been registered for that SQL query. If a query runs a single job, this isn't an issue: in that case, all {{LiveStageMetrics}} instances will hold the same immutable {{{}accumIdsToMetricType{}}}.

The problem arises if we have a query that runs many jobs (e.g. a complex query with many joins which gets divided into many jobs due to AQE): in that case, each job submission results in a new {{accumIdsToMetricType}} map being created.

This PR fixes this by changing {{accumIdsToMetricType}} to be a mutable {{ConcurrentHashMap}} which is shared across all {{LivestageMetrics}} instances belonging to the same {{{}LiveExecutionData{}}}.

The modified classes are {{private}} and are used only in SQLAppStatusListener, so I don't think this change poses any realistic risk of binary incompatibility risks to third party code.
Bug: Replace `scala.runtime.Tuple2Zipped` to `scala.collection.LazyZip2`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/23 08:30
Affects Version/s: 4.0.0


{code:java}
@scala.deprecated(message = "Use scala.collection.LazyZip2.", since = "2.13.0")
final class Tuple2Zipped[El1, It1 <: scala.Iterable[El1], El2, It2 <: scala.Iterable[El2]](colls : scala.Tuple2[It1, It2]) extends scala.AnyVal with scala.runtime.ZippedIterable2[El1, El2] {

@scala.deprecated(message = "Use scala.collection.LazyZip2.", since = "2.13.0")
object Tuple2Zipped extends scala.AnyRef {
  final class Ops[T1, T2](x : scala.Tuple2[T1, T2]) extends scala.AnyVal {
    @scala.deprecated(message = "Use xs.lazyZip(yz).map((_, _))", since = "2.13.0")
    def invert[El1, It1[a] <: scala.Iterable[a], El2, It2[a] <: scala.Iterable[a], That](implicit w1 : scala.<:<[T1, It1[El1]], w2 : scala.<:<[T2, It2[El2]], bf : scala.collection.BuildFrom[T1, scala.Tuple2[El1, El2], That]) : That = { /* compiled code */ }
    @scala.deprecated(message = "Use xs.lazyZip(ys)", since = "2.13.0")
    def zipped[El1, It1 <: scala.Iterable[El1], El2, It2 <: scala.Iterable[El2]](implicit w1 : scala.Function1[T1, scala.collection.IterableOps[El1, scala.Iterable, It1] with It1], w2 : scala.Function1[T2, scala.collection.IterableOps[El2, scala.Iterable, It2] with It2]) : scala.runtime.Tuple2Zipped[El1, It1, El2, It2] = { /* compiled code */ }
  }
{code}
Bug: Replace: _LEGACY_ERROR_TEMP_3007
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/23 15:40
Affects Version/s: 4.0.0

def checkpointRDDBlockIdNotFoundError(rddBlockId: RDDBlockId): Throwable = \{
    new SparkException(
      errorClass = "_LEGACY_ERROR_TEMP_3007",
      messageParameters = Map("rddBlockId" -> s"$rddBlockId"),
      cause = null
    )
  }
This error condition appears to be quite common, so we should convert it to a proper error class.
Bug: Add "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" so Platform can access cleaner on Java 9+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/23 00:59
Affects Version/s: 3.3.2

We need to add `--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED` to our JVM options so that the code in `org.apache.spark.unsafe.Platform` can access the JDK internal cleaner classes.
Bug: Support stage level task resource profile for k8s cluster when dynamic allocation disabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/23 00:38
Affects Version/s: 3.4.1

[https://github.com/apache/spark/pull/37268] has introduced a new feature that supports stage-level schedule task resource profile for standalone cluster when dynamic allocation is disabled. It's really cool feature, especially for ML/DL cases, more details can be found in that PR.

 

The problem here is that the feature is only available for standalone and YARN cluster for now, but most users would also expect it can be used for other spark clusters like K8s.

 

So I filed this issue to track this task.
Bug: XML: Add support for value in 'rowTag' element
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/23 19:04
Affects Version/s: 4.0.0

The following XML with rowTag 'book' will yield a schema with just "_id" column and not the value:
 
{code:java}
 <p><book id="1">Great Book</book> </p>   {code}
Let's parse value as well.
Bug: Replace: _LEGACY_ERROR_TEMP_2151 with a proper error class
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 10/Oct/23 22:19
Affects Version/s: 4.0.0

{code:java}
def expressionDecodingError(e: Exception, expressions: Seq[Expression]): SparkRuntimeException = {
  new SparkRuntimeException(
    errorClass = "_LEGACY_ERROR_TEMP_2151",
    messageParameters = Map(
      "e" -> e.toString(),
      "expressions" -> expressions.map(
        _.simpleString(SQLConf.get.maxToStringFields)).mkString("\n")),
    cause = e)
} {code}
Bug: Replace `toIterator` with `iterator` for `IterableOnce`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Oct/23 11:12
Affects Version/s: 4.0.0

{code:java}
@deprecated("Use .iterator instead", "2.13.0")
@`inline` def toIterator: Iterator[A] = it.iterator {code}
Bug: Replace: _LEGACY_ERROR_TEMP_2187 with a better error message
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/23 22:24
Affects Version/s: 4.0.0

{code:java}
def convertHiveTableToCatalogTableError(
    e: SparkException, dbName: String, tableName: String): Throwable = {
  new SparkException(
    errorClass = "_LEGACY_ERROR_TEMP_2187",
    messageParameters = Map(
      "message" -> e.getMessage,
      "dbName" -> dbName,
      "tableName" -> tableName),
    cause = e)
} {code}
Bug: Replace: _LEGACY_ERROR_TEMP_2152 with a better error class
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/23 22:23
Affects Version/s: 4.0.0

{code:java}
def expressionEncodingError(e: Exception, expressions: Seq[Expression]): SparkRuntimeException = {
  new SparkRuntimeException(
    errorClass = "_LEGACY_ERROR_TEMP_2152",
    messageParameters = Map(
      "e" -> e.toString(),
      "expressions" -> expressions.map(
        _.simpleString(SQLConf.get.maxToStringFields)).mkString("\n")),
    cause = e)
} {code}
Bug: Replace:   _LEGACY_ERROR_TEMP_2134 with a regular error class
Status: Open | Priority: Major | Resolution: 
Created: 10/Oct/23 22:15
Affects Version/s: 4.0.0

This is a frequently seen error we should convert:
def cannotParseStringAsDataTypeError(pattern: String, value: String, dataType: DataType)
: SparkRuntimeException = {
new SparkRuntimeException(
errorClass = "_LEGACY_ERROR_TEMP_2134",
messageParameters = Map(
"value" -> toSQLValue(value),
"pattern" -> toSQLValue(pattern),
"dataType" -> dataType.toString))
}
Bug: Recover Python build in branch-3.3, 3.4 and 3.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/23 08:31
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/workflows/build_branch33.yml
https://github.com/apache/spark/actions/workflows/build_branch34.yml
https://github.com/apache/spark/actions/workflows/build_branch35.yml
Bug: Cache Invalidation Issue with JDBC Table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/23 03:36
Affects Version/s: 3.5.0

We have identified a cache invalidation issue when caching JDBC tables in Spark SQL. The cached table is unexpectedly invalidated when queried, leading to a re-read from the JDBC table instead of retrieving data from the cache.
Example SQL:

{code:java}
CACHE TABLE cache_t SELECT * FROM mysql.test.test1;
SELECT * FROM cache_t;
{code}

Expected Behavior:
The expectation is that querying the cached table (cache_t) should retrieve the result from the cache without re-evaluating the execution plan.

Actual Behavior:
However, the cache is invalidated, and the content is re-read from the JDBC table.

Root Cause:
The issue lies in the 'CacheData' class, where the comparison involves 'JDBCTable.' The 'JDBCTable' is a case class:
{code:java}
case class JDBCTable(ident: Identifier, schema: StructType, jdbcOptions: JDBCOptions)
{code}

The comparison of non-case class components, such as 'jdbcOptions,' involves pointer comparison. This leads to unnecessary cache invalidation.
Bug: Avoid paste string value of hive orc compression kind
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Oct/23 12:00
Affects Version/s: 4.0.0

Currently, hive supports orc format with some compression codec( Please refer org.apache.hadoop.hive.ql.io.orc.CompressionKind).
Spark pasted many string name of these compression codec.
Bug: Revisit TableCacheQueryStage to avoid replicated InMemoryRelation materialization
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/23 22:00
Affects Version/s: 3.5.0

TableCacheQueryStage is created per InMemoryTableScanExec by AdaptiveSparkPlanExec and it materializes InMemoryTableScanExec output (cached RDD) to provide runtime stats in order to apply AQE  optimizations into remaining physical plan stages. TableCacheQueryStage materializes InMemoryTableScanExec eagerly by submitting job per TableCacheQueryStage instance. For example, if there are 2 TableCacheQueryStage instances referencing same IMR instance (cached RDD) and first InMemoryTableScanExec' s materialization takes longer, following logic will return false (inMemoryTableScan.isMaterialized => false) and this may cause replicated IMR materialization. This behavior can be more visible when cached RDD size is high.
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala#L281]

Would like to get community feedback. Thanks in advance.
cc [~ulysses] [~cloud_fan]

*Sample Query to simulate the problem:*
// Both join legs uses same IMR instance
{code:java}
import spark.implicits._

val arr = (1 to 12).map { i => {
    val index = i % 5
    (index, s"Employee_$index", s"Department_$index")
  }
}
val df = arr.toDF("id", "name", "department")
  .filter('id >= 0)
  .sort("id")
  .groupBy('id, 'name, 'department)
  .count().as("count")
df.persist()

val df2 = df.sort("count").filter('count <= 2)
val df3 = df.sort("count").filter('count >= 3)
val df4 = df2.join(df3, Seq("id", "name", "department"), "fullouter")

df4.show() {code}
*Physical Plan:*
{code:java}
== Physical Plan ==
AdaptiveSparkPlan (31)
+- == Final Plan ==
   CollectLimit (21)
   +- * Project (20)
      +- * SortMergeJoin FullOuter (19)
         :- * Sort (10)
         :  +- * Filter (9)
         :     +- TableCacheQueryStage (8), Statistics(sizeInBytes=210.0 B, rowCount=5)
         :        +- InMemoryTableScan (1)
         :              +- InMemoryRelation (2)
         :                    +- AdaptiveSparkPlan (7)
         :                       +- HashAggregate (6)
         :                          +- Exchange (5)
         :                             +- HashAggregate (4)
         :                                +- LocalTableScan (3)
         +- * Sort (18)
            +- * Filter (17)
               +- TableCacheQueryStage (16), Statistics(sizeInBytes=210.0 B, rowCount=5)
                  +- InMemoryTableScan (11)
                        +- InMemoryRelation (12)
                              +- AdaptiveSparkPlan (15)
                                 +- HashAggregate (14)
                                    +- Exchange (13)
                                       +- HashAggregate (4)
                                          +- LocalTableScan (3) {code}
*Stages DAGs materializing the same IMR instance:*
!IMR Materialization - Stage 2.png|width=303,height=134!
!IMR Materialization - Stage 3.png|width=303,height=134!
Bug: Avoid reusing rocksdb sst files in a dfferent rocksdb instance by removing file version map entry of larger versions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/23 00:38
Affects Version/s: 3.5.0

When loading a rocksdb instance, remove file version map entry of larger versions to avoid rocksdb sst file unique id mismatch exception. The SST files in larger versions can't be reused even if they have the same size and name because they belong to another rocksdb instance.
Bug: Regression in CSV schema inference when timestamps do not match specified timestampFormat
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/23 17:25
Affects Version/s: 3.5.0

There is a regression in Spark 3.5.0 when inferring the schema of CSV files containing timestamps, where a column will be inferred as a timestamp even if the contents do not match the specified timestampFormat.

*Test Data*

I have the following CSV file:
{code:java}
2884-06-24T02:45:51.138
2884-06-24T02:45:51.138
2884-06-24T02:45:51.138
{code}
*Spark 3.4.0 Behavior (correct)*

In Spark 3.4.0, if I specify the correct timestamp format, then the schema is inferred as timestamp:
{code:java}
scala> val df = spark.read.option("timestampFormat", "yyyy-MM-dd'T'HH:mm:ss.SSS").option("inferSchema", true).csv("/tmp/timestamps.csv")
df: org.apache.spark.sql.DataFrame = [_c0: timestamp]
{code}
If I specify an incompatible timestampFormat, then the schema is inferred as string:
{code:java}
scala> val df = spark.read.option("timestampFormat", "yyyy-MM-dd'T'HH:mm:ss").option("inferSchema", true).csv("/tmp/timestamps.csv")
df: org.apache.spark.sql.DataFrame = [_c0: string]
{code}
*Spark 3.5.0*

In Spark 3.5.0, the column will be inferred as timestamp even if the data does not match the specified timestampFormat.
{code:java}
scala> val df = spark.read.option("timestampFormat", "yyyy-MM-dd'T'HH:mm:ss").option("inferSchema", true).csv("/tmp/timestamps.csv")
df: org.apache.spark.sql.DataFrame = [_c0: timestamp]
{code}
Reading the DataFrame then results in an error:
{code:java}
Caused by: java.time.format.DateTimeParseException: Text '2884-06-24T02:45:51.138' could not be parsed, unparsed text found at index 19
{code}
Bug: Introduce a mapper for StorageLevel
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/23 10:37
Affects Version/s: 4.0.0

Currently, StorageLevel provides fromString to get the StorageLevel's instance with StorageLevel's name. So developers or users have to copy the string literal about StorageLevel's name to set or get instance of StorageLevel. This issue lead to developers need to manually maintain its consistency. It is easy to make mistakes and reduce development efficiency.
Bug: Add a new method `cleanup` in the UDTF interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Oct/23 20:45
Affects Version/s: 3.5.0

Currently, the {{terminate}} method of a UDTF is always executed, regardless of whether the {{eval}} method calls are successful. This is problematic. We should execute {{terminate}} only when all {{eval}} calls succeed.

But what if users wish to perform cleanup actions during UDTF execution, such as closing connections? One option is for users to embed a {{try...except}} logic within the {{eval}} call:
{code:java}
def eval(self, row: Any):
  try:
    run_code()
  except Exception:
    clean_up(){code}
However, running this {{try...except}} block for every {{eval}} call can be expensive to run, potentially affecting the performance of UDTFs.

To tackle this, we can introduce a new method in the UDTF interface that will be called regardless of the outcome. The logic would look like:
{code:java}
try:
  eval()
  terminate()
finally:
  cleanup(){code}
Bug: Surface sc.setLocalProperty() value=NULL param meaning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/23 19:41
Affects Version/s: 3.5.0

sc.setLocalProperty() has special meaning/feature and when supplied then removes the property associated to key parameter. It's only mentioned in the Fair Scheduler section of the documentation.

It would be nice to document that on the API's as well for users.
Bug: Incorrect summary counts from a CSV file
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/23 20:51
Affects Version/s: 3.5.0

I am using pip-installed Pyspark version 3.5.0 inside the context of an IPython shell. The task is straightforward: take [this CSV file|https://gist.githubusercontent.com/evanvolgas/e5cb082673ec947239658291f2251de4/raw/a9c5e9866ac662a816f9f3828a2d184032f604f0/AAPL.csv] of AAPL stock prices and compute the minimum and maximum volume weighted average price for the entire file. 

My code is [here. |https://gist.github.com/evanvolgas/e4aa75fec4179bb7075a5283867f127c]I've also performed the same computation in DuckDB because I noticed that the results of the Spark code are wrong. 

Literally, the exact same SQL in DuckDB and in Spark yield different results, and Spark's are wrong. 

I have never seen this behavior in a Spark release before. I'm very confused by it, and curious if anyone else can replicate this behavior.
Bug: Decimal precision exceeds max precision error when using unary minus on min Decimal values on Scala 2.13 Spark
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/23 17:46
Affects Version/s: 3.2.0

When submitting an application to Spark built with Scala 2.13, there are issues with Decimal overflow that show up when using unary minus (and also {{abs()}} which uses unary minus under the hood.

Here is an example PySpark reproduce use case:

{code}
from decimal import Decimal

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType,StructField, DecimalType

spark = SparkSession.builder \
      .master("local[*]") \
      .appName("decimal_precision") \
      .config("spark.rapids.sql.explain", "ALL") \
      .config("spark.sql.ansi.enabled", "true") \
      .config("spark.sql.legacy.allowNegativeScaleOfDecimal", 'true') \
      .getOrCreate()  

precision = 38
scale = 0
DECIMAL_MIN = Decimal('-' + ('9' * precision) + 'e' + str(-scale))

data = [[DECIMAL_MIN]]

schema = StructType([
    StructField("a", DecimalType(precision, scale), True)])
df = spark.createDataFrame(data=data, schema=schema)

df.selectExpr("a", "-a").show()
{code}

This particular example will run successfully on Spark built with Scala 2.12, but throw a java.math.ArithmeticException on Spark built with Scala 2.13. 

If you change the value of {{DECIMAL_MIN}} in the previous code to something just ahead of the original DECIMAL_MIN, you will not get an exception thrown, but instead you will get an incorrect answer (possibly due to overflow):

{code}
...
DECIMAL_MIN = Decimal('-8' + ('9' * (precision-1)) + 'e' + str(-scale))
...
{code} 

Output:
{code}
+--------------------+--------------------+
|                   a|               (- a)|
+--------------------+--------------------+
|-8999999999999999...|90000000000000000...|
+--------------------+--------------------+
{code}

It looks like the code in {{Decimal.scala}} uses {{scala.math.BigDecimal}}. See https://github.com/scala/bug/issues/11590 with updates on how Scala 2.13 handles BigDecimal. It looks like there is {{java.math.MathContext}} missing when performing these operations.
Bug: Upgrade SNAPPY to 1.1.10.5 to pick up fix re Linux PLE64
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/23 13:43
Affects Version/s: 3.5.0

SPARK-45323 move to Snappy 1.1.10.4 and is proposing to add to SPARK 3.5.1

Snappy prior to 1.1.10.5 will not work on Linux PLE 64.

Moving to Snappy 1.1.10.5 will address that issue
https://github.com/xerial/snappy-java/pull/515
Bug: Update Partition Stats
Status: Open | Priority: Major | Resolution: 
Created: 05/Oct/23 10:33
Affects Version/s: 3.5.0

spark brought *spark.sql.statistics.size.autoUpdate.enabled* and it is a good feature for small tables or tables does not contains a lot of files.
It would be great also to introduce a new option that calculates statistics on the partition level. In other words, Instead of altering/updating the whole table statistics, it only gathers the statistics of the partitions that spark writes to the table.