av9ash/GitBugs huggingface dataset:
Bug: Add Imputer inputCols validation for empty input case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/22 07:50
Affects Version/s: 3.3.0

If Imputer inputCols is empty, the `fit` works fine but when saving model, error will be raised:
{quote}AnalysisException: 
Datasource does not support writing empty or nested empty schemas.
Please make sure the data schema has at least one or more column(s).
{quote}
Bug: Extend new heartbeat mechanism to YARN
Status: Open | Priority: Major | Resolution: 
Created: 13/Aug/22 04:16
Affects Version/s: 3.4.0

Extend the new heartbeat mechanism in SPARK-39984 to YARN.

 

SPARK-39984 issue:

[https://issues.apache.org/jira/projects/SPARK/issues/SPARK-39984?filter=allopenissues]

 

SPARK-39984 PR:

[https://github.com/apache/spark/pull/37411]
Bug: Extend the new heartbeat mechanism to Kubernetes
Status: Open | Priority: Major | Resolution: 
Created: 13/Aug/22 04:18
Affects Version/s: 3.4.0

Extend the new heartbeat mechanism in SPARK-39984 to Kubernetes.

 

SPARK-39984 issue:

[https://issues.apache.org/jira/projects/SPARK/issues/SPARK-39984?filter=allopenissues]

 

SPARK-39984 PR:

[https://github.com/apache/spark/pull/37411]
Bug: Restore the error handling syntax of try_cast()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/22 05:00
Affects Version/s: 3.4.0

For the following query
{code:java}
SET spark.sql.ansi.enabled=true;
SELECT try_cast(1/0 AS string); {code}
Spark 3.3 will throw an exception for the division by zero error. In current master branch, it returns null after the refactoring PR https://github.com/apache/spark/pull/36703
Bug: Row indexes can overshadow user-created data
Status: Open | Priority: Major | Resolution: 
Created: 12/Aug/22 15:11
Affects Version/s: 3.4.0

https://github.com/apache/spark/pull/37228 introduces ability to compute row indexes, which users can access through `_metadata.row_index` column. Internally this is achieved with the help of an extra column `_tmp_metadata_row_index`. When present in the schema sent to parquet reader, the reader populates it with row indexes, and the values are later placed in the `_metadata` struct. 

While relatively unlikely, it's still possible, that a user might want to include column `_tmp_metadata_row_index` in their data. In such scenario, the column will be populated with row indexes, rather than data read from the file.

For repro, search `FileMetadataStructRowIndexSuite.scala` for this Jira ticket number.

We could introduce some kind of countermeasure to handle this scenario.
Bug: Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/22 21:34
Affects Version/s: 3.4.0

There are two possible reasons, including Network Failure and Task Failure, to make RPC failures.

(1) Task Failure: The network is good, but the task causes the executor's JVM crash. Hence, RPC fails.
(2) Network Failure: The executor works well, but the network between Driver and Executor is broken. Hence, RPC fails.

We should handle these two different kinds of failure in different ways. First, if the failure is Task Failure, we should increment the variable `{{{}numFailures`{}}}. If the value of {{`numFailures`}} is larger than a threshold, Spark will label the job failed. Second, if the failure is Network Failure, we will not increment the variable `{{{}numFailures`{}}}. We will just assign the task to a new executor. Hence, the job will not be recognized as failed due to Network Failure.

However, currently, Spark recognizes every RPC failure as Task Failure. Hence, it will cause extra Spark job failures.
Bug: Incorrect target interval type in cast overflow errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/22 05:32
Affects Version/s: 3.4.0

The example below shows the issue:

{code:sql}
spark-sql> select CAST(9223372036854775807L AS INTERVAL YEAR TO MONTH);
org.apache.spark.SparkArithmeticException: [CAST_OVERFLOW] The value 9223372036854775807L of the type "BIGINT" cannot be cast to "INTERVAL MONTH" due to an overflow.
{code}

The target type "INTERVAL MONTH" is incorrect.
Bug: spark-shell can not instantiate object whose class is defined in REPL dynamically
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/22 02:47
Affects Version/s: 2.4.8

spark-shell throws {{NoSuchMethodException}} if I define a class in REPL and then call {{newInstance}} via reflection.

{code:scala}
Spark context available as 'sc' (master = yarn, app id = application_1656488084960_0162).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.3
      /_/
         
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_141)
Type in expressions to have them evaluated.
Type :help for more information.

scala> class Demo {
     |   def demo(s: String): Unit = println(s)
     | }
defined class Demo

scala> classOf[Demo].newInstance().demo("OK")
java.lang.InstantiationException: Demo
  at java.lang.Class.newInstance(Class.java:427)
  ... 47 elided
Caused by: java.lang.NoSuchMethodException: Demo.<init>()
  at java.lang.Class.getConstructor0(Class.java:3082)
  at java.lang.Class.newInstance(Class.java:412)
  ... 47 more
{code}

But the same code works fine in native scala REPL:

{code:scala}
Welcome to Scala 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation. Or try :help.

scala> class Demo {
     |   def demo(s: String): Unit = println(s)
     | }
defined class Demo

scala> classOf[Demo].newInstance().demo("OK")
OK
{code}

What's the difference between spark-shell REPL and native scala REPL?

I guess the Demo class might be treated as inner class in spark-shell REPL.

But ... how to solve the problem?
Bug: Support Decimal128 type
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/22 07:04
Affects Version/s: 3.4.0

Spark SQL today supports the DECIMAL data type. The implementation of Decimal that can hold a BigDecimal or Long.  Decimal provides some operators like +, -, *, / and so on.
Take the + as example, the implementation show below.

{code:java}
  def + (that: Decimal): Decimal = {
    if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
      Decimal(longVal + that.longVal, Math.max(precision, that.precision) + 1, scale)
    } else {
      Decimal(toBigDecimal.bigDecimal.add(that.toBigDecimal.bigDecimal))
    }
  }
{code}

We can see there exists two addition and call Decimal.apply. The add operator of BigDecimal will construct a new BigDecimal instance.
The implementation of Decimal.apply will call new to construct a new Decimal instance with the new BigDecimal instance.
As we know, Decimal instance will hold the new BigDecimal instance.
If a large table has a Decimal field called 'colA, the execution of SUM('colA) will involve the creation of a large number of Decimal instances and BigDecimal instances. These Decimal instances and BigDecimal instances will lead to garbage collection to occur frequently.

Decimal128 is a high-performance decimal about 8X more efficient than Java BigDecimal for typical operations. It uses a finite (128 bit) precision and can handle up to decimal(38, X). It is also "mutable" so you can change the contents of an existing object. This helps reduce the cost of new() and garbage collection.

We have generate a benchmark report for compare Spark Decimal, Java BigDecimal and Decimal128. Please see the attachment.

In this new feature, we will introduce DECIMAL128 to accelerate decimal calculation.

h3. Milestone 1 – Spark Decimal equivalency ( The new Decimal type Decimal128 meets or exceeds all function of the existing SQL Decimal):
* Add a new DataType implementation for Decimal128.
* Support Decimal128 in Dataset/UDF.
* Decimal128 literals
* Decimal128 arithmetic(e.g. Decimal128 + Decimal128, Decimal128 - Decimal)
* Decimal or Math functions/operators: POWER, LOG, Round, etc
* Cast to and from Decimal128, cast String/Decimal to Decimal128, cast Decimal128 to string (pretty printing)/Decimal, with the * * SQL syntax to specify the types
* Support sorting Decimal128.

h3. Milestone 2 – Persistence:
 * Ability to create tables of type Decimal128
 * Ability to write to common file formats such as Parquet and JSON.
 * INSERT, SELECT, UPDATE, MERGE
 * Discovery

h3. Milestone 3 – Client support
 * JDBC support
 * Hive Thrift server

h3. Milestone 4 – PySpark and Spark R integration
 * Python UDF can take and return Decimal128
 * DataFrame support
Bug: Replace postgresql 42.3.3 with 42.2.26
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 06/Aug/22 21:15
Affects Version/s: 3.4.0

postgresql >= 42.3.0, < 42.4.1 is affected by [CVE-2022-31197|https://github.com/pgjdbc/pgjdbc/security/advisories/GHSA-r38f-c4h4-hqq2]

|
Upgrade postgresql to 42.4.1 won't pass Github actions tests. 

[42.2.26|https://github.com/pgjdbc/pgjdbc/commits/release/42.2] is a backport to fix this CVE
Bug: LevelDBIterator not close after used in `RemoteBlockPushResolver`, `YarnShuffleService` and `ExternalShuffleBlockResolver` 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/22 05:00
Affects Version/s: 3.4.0

For example:
{code:java}
@VisibleForTesting
static ConcurrentMap<AppExecId, ExecutorShuffleInfo> reloadRegisteredExecutors(DB db)
    throws IOException {
  ConcurrentMap<AppExecId, ExecutorShuffleInfo> registeredExecutors = Maps.newConcurrentMap();
  if (db != null) {
    DBIterator itr = db.iterator();
    itr.seek(APP_KEY_PREFIX.getBytes(StandardCharsets.UTF_8));
    while (itr.hasNext()) {
      Map.Entry<byte[], byte[]> e = itr.next();
      String key = new String(e.getKey(), StandardCharsets.UTF_8);
      if (!key.startsWith(APP_KEY_PREFIX)) {
        break;
      }
      AppExecId id = parseDbAppExecKey(key);
      logger.info("Reloading registered executors: " +  id.toString());
      ExecutorShuffleInfo shuffleInfo = mapper.readValue(e.getValue(), ExecutorShuffleInfo.class);
      registeredExecutors.put(id, shuffleInfo);
    }
  }
  return registeredExecutors;
} {code}
DBIterator in above code should call close after used, otherwise, there is no other place to close it
Bug: ParquetSchemaConverter fails match schema by id
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Aug/22 13:31
Affects Version/s: 3.3.0


{code:scala}
  test("SPARK-38094: absence of field ids: reading nested schema struct field renamed") {
    withTempDir { dir =>
      // now with nested schema/complex type

      val innerTypeRenamed = new StructType().add("c1", IntegerType, true, withId(6));
      val readSchema =
        new StructType()
          .add("c", ArrayType(innerTypeRenamed), true, withId(3))
          .add("e", IntegerType, true, withId(5))


      val innerType = new StructType().add("c0", IntegerType, true, withId(6))
      val writeSchema =
        new StructType()
          .add("c", ArrayType(innerType), true, withId(3))
          .add("randomName", StringType, true)

      val writeData = Seq(Row(Seq(Row(100)), "text"), Row(Seq(Row(100)), "more"))

      spark.createDataFrame(writeData.asJava, writeSchema)
        .write.mode("overwrite").parquet(dir.getCanonicalPath)

      withAllParquetReaders {
        checkAnswer(spark.read.schema(readSchema).parquet(dir.getCanonicalPath),
          // a, b, c, d all couldn't be found
          Row(Seq(Row(100)), null) :: Row(Seq(Row(100)), null) :: Nil)
      }
    }
  }
{code}
Bug:  Restrict special characters in field name, which can be controlled by switches
Status: Open | Priority: Major | Resolution: 
Created: 05/Aug/22 13:08
Affects Version/s: 3.4.0

The hive metastore restricts field name to only contain alphanumerics and underscores. If the custom catalog does not use hms, Custom metadata system based on iceberg。these restrictions may not exist, such as reading excel data, writing iceberg table.

hack way forbidden：
{code:java}
@Around("execution(public * org.apache.spark.sql.execution.datasources.DataSourceUtils.checkFieldNames(..))")
public void checkFieldNames_1(ProceedingJoinPoint pjp) throws Throwable {
LOG.info("skip checkFieldNames 1");
}

@Around("execution(public * org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$.checkFieldNames(..))")
public void checkFieldNames_2(ProceedingJoinPoint pjp) throws Throwable {
LOG.info("skip checkFieldNames 2");
}

@Around("execution(public * org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$.checkFieldName(..))")
public void checkFieldNames_3(ProceedingJoinPoint pjp) throws Throwable
{ LOG.info("skip checkFieldNames 3"); }{code}
CREATE OR REPLACE TABLE huaixin_rp.bigdata.parquet_orders_rp5 USING ICEBERG  select 12 as id, 'hangzhou' as `address(地  址)`

[~hyukjin.kwon]
Bug: Check workerLastHeartbeat with master before HeartbeatReceiver expires an executor
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Aug/22 22:40
Affects Version/s: 3.4.0

Currently, the driver’s HeartbeatReceiver will expire an executor if it does not receive any heartbeat from the executor for 120 seconds. However, 120 seconds is too long, but we will face other challenges when we try to lower the timeout threshold. To elaborate, when an executor is performing GC, it cannot reply any message.

 

Hence, this PR aims to provide a method to lower the timeout. Worker will send heartbeats to master periodically, and thus if HeartbeatReceiver asks master the information about the latest heartbeat from the worker which the executor is on, HeartbeatReceiver can determine whether the heartbeat loss is caused by network issues or other issues (e.g. GC). If the heartbeat loss is not caused by network issues, the HeartbeatReceiver will put the executor into a waitingList rather than expiring it immediately.
Bug: Introduce ThrottledLogger to prevent log message flooding caused by network issues
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Aug/22 22:42
Affects Version/s: 3.4.0

When transient network error occurs, Spark may write out a large volume of error logs related to the network errors. The excessive logging can create further problems downstream, e.g. blow up our log storage. If we can combine / batch the network errors when they come in a burst, we only print out periodic summaries of errors that are happening repeatedly in a short time window.
Bug: Make filtered distinct count more accurate
Status: Open | Priority: Major | Resolution: 
Created: 04/Aug/22 10:25
Affects Version/s: 3.4.0

How to reproduce this issue:
{code:scala}
spark.sql("set spark.sql.cbo.enabled=true")
spark.sql("CREATE TABLE date_dim(d_month_seq int, d_year int, d_qoy int, d_moy int) USING parquet LOCATION 'file:/Users/yumwang/data/date_dim'")
spark.sql("analyze table date_dim compute statistics for all columns")
spark.sql("select * from date_dim where d_month_seq >= 1212 and d_month_seq <= 1223").explain("cost")
{code}

Output attribute stats:
{code:java}
case class Statistics(
    sizeInBytes: BigInt,
    rowCount: Option[BigInt] = None,
    attributeStats: AttributeMap[ColumnStat] = AttributeMap(Nil),
    isRuntime: Boolean = false) {

  override def toString: String = "Statistics(" + simpleString + ")"

  /** Readable string representation for the Statistics. */
  def simpleString: String = {
    Seq(s"sizeInBytes=${Utils.bytesToString(sizeInBytes)}",
      if (rowCount.isDefined) {
        // Show row count in scientific notation.
        s"rowCount=${BigDecimal(rowCount.get, new MathContext(3, RoundingMode.HALF_UP)).toString()}, attributeStats=${attributeStats}"
      } else {
        ""
      }
    ).filter(_.nonEmpty).mkString(", ")
  }
}
{code}
Bug: Principals in KafkaTestUtils should use canonical host name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/22 08:22
Affects Version/s: 3.4.0

We figured out KafkaDelegationTokenSuite is failing on AWS EC2.

During investigation we found the root cause as SPARK-39530 - from the debug message of kerberos authentication, the host part of principal seems to be required as "canonical host name".

Before SPARK-39530, the test used the IP as "127.0.0.1" (loopback) and the host part of principal as the canonical host name of "127.0.0.1".

While it is still valid to pick up the IP rather than loopback, we still need to make sure canonical host name is used in principal.
Bug: Determine task failures based on ExecutorExitCode
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Aug/22 00:51
Affects Version/s: 3.4.0

There are a lot of possible reasons to cause an executor exit. However, the driver will assume every executor exit is caused by task failure. The assumption is wrong. For example, when DiskBlockManager fails to create a directory, it will close executor’s JVM with the exit code {{{}DISK_STORE_FAILED_TO_CREATE_DIR{}}}. Obviously, when the driver received the exit code {{{}DISK_STORE_FAILED_TO_CREATE_DIR{}}}, the executor exit is highly possible caused by hardware failure rather than task failure.
Bug: Batch query cannot read the updates from streaming query if streaming query writes to the catalog table via DSv1 sink
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/22 02:13
Affects Version/s: 3.2.3

(I think this should be ancient issue but there's no good way to list up all affected versions, so I just pick up the recent version in each version line.)

When streaming query writes to catalog table via DSv1 sink, there is no refreshing/invalidation of the destination table, hence querying the destination table with batch query is not guaranteed to read the latest "committed" updates.
Bug: Cache Hints
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Aug/22 08:20
Affects Version/s: 3.4.0

This ticket intends to add query hints for cache behaviors, user can perform actions like the use/skip/cache/uncache, etc on the cached results by the following hints.

 

- RESULT_CACHE

 spark will use cache by default, this hint keep this behavior but will cache the child plan if uncached yet.

- NO_RESULT_CACHE

with this hint, we can skip the default behavior to by pass the cached data if it is cache

- RESULT_UNCACHE

with this hint, we can remove the cached data.

 

RESULT_CACHE and NO_RESULT_CACHE are borrowed from oracle, FYI,

[https://docs.oracle.com/database/121/TGDBA/tune_result_cache.htm#TGDBA647.|https://docs.oracle.com/database/121/TGDBA/tune_result_cache.htm#TGDBA647]

RESULT_UNCACHE's added accordingly.

 

Similar ideas can also be found in MySQL, FYI [https://dev.mysql.com/doc/refman/5.7/en/query-cache-in-select.html]

 

Maybe we can add more cache hints like RESULT_RECACHE to refresh the cached data, or maybe some cache behaviors that relate to metadata.
Bug: Should not cache unserialized broadcast relations on the driver
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Aug/22 20:55
Affects Version/s: 3.3.0

In TorrentBroadcast.writeBlocks we store the unserialized broadcast object in addition to the serialized version of it - 
{code:java}
private def writeBlocks(value: T): Int = {
    import StorageLevel._
    // Store a copy of the broadcast variable in the driver so that tasks run on the driver
    // do not create a duplicate copy of the broadcast variable's value.
    val blockManager = SparkEnv.get.blockManager
    if (!blockManager.putSingle(broadcastId, value, MEMORY_AND_DISK, tellMaster = false)) {
      throw new SparkException(s"Failed to store $broadcastId in BlockManager")
    }
 {code}
In case of broadcast relations, these objects can be fairly large (60MB in one observed case) and are not strictly necessary on the driver.

Add the option to not keep the unserialized versions of the objects.
Bug: Upgrade com.google.crypto.tink:tink from 1.6.1 to 1.7.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Aug/22 17:26
Affects Version/s: 3.4.0

[CVE-2022-25647|https://www.cve.org/CVERecord?id=CVE-2022-25647]

[Info at SNYK|https://security.snyk.io/vuln/SNYK-JAVA-COMGOOGLECODEGSON-1730327]

[CVE-2021-22569|https://www.cve.org/CVERecord?id=CVE-2021-22569]

[Info at SNYK|https://security.snyk.io/vuln/SNYK-JAVA-COMGOOGLEPROTOBUF-2331703]

[releases log|https://github.com/google/tink/releases/tag/v1.7.0]
Bug: PathOutputCommitters to work with dynamic partition overwrite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Aug/22 14:24
Affects Version/s: 3.5.0

sibling of MAPREDUCE-7403: allow PathOutputCommitter implementation to declare that they support the semantics required by spark dynamic partitioning:

* rename to work as expected
* working dir to be on same fs as final dir

They will do this through implementing StreamCapabilities and adding a new probe, "mapreduce.job.committer.dynamic.partitioning" ; the spark side changes are to
* postpone rejection of dynamic partition overwrite until the output committer is created
* allow it if the committer implements StreamCapabilities and returns true for {{hasCapability("mapreduce.job.committer.dynamic.partitioning")))

this isn't going to be supported by the s3a committers, they don't meet the requirements. The manifest committer of MAPREDUCE-7341 running against abfs and gcs does work.
Bug: Spark AWS SDK and kinesis dependencies lagging hadoop-aws and s3a
Status: Open | Priority: Minor | Resolution: 
Created: 03/Aug/22 13:04
Affects Version/s: 3.4.0

The AWS SDK and matching kinesis versions are now a few iterations behind what is shipping in hadoop 3.3.x. ( see HADOOP-18068 and HADOOP-18344)

* this updates dependencies/bundling of jackson and httpclient
* no problems upgrading other than some test regressions

catching up would be good, as it means that recent s3a releases are not qualified with the AWS SDK release spark is pulling in -and if there is any problem. it'll be a spark team issue.
Bug: Remove useless configuration & extract common code for parquet read
Status: In Progress | Priority: Minor | Resolution: 
Created: 21/Aug/22 01:58
Affects Version/s: 3.4.0

# Remove useless configuration: hadoopConf.set(ParquetWriteSupport.SPARK_ROW_SCHEMA, readDataSchemaAsJson)
 # extract common code:

{quote}ParquetFileFormat.buildReaderWithPartitionValues ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L202-L228])
parquet/ParquetScan.createReaderFactory ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScan.scala#L66-L93])
{quote}
Bug: Add helper function for DL model inference in pyspark.ml.functions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Aug/22 23:48
Affects Version/s: 3.2.2

Add a helper function to create a pandas_udf for inference on a given DL model, where the user provides a predict function that is responsible for loading the model and inferring on a batch of numpy inputs.
Bug: pyspark.pandas .apply() changing rows ordering
Status: Open | Priority: Minor | Resolution: 
Created: 12/Aug/22 21:10
Affects Version/s: 3.3.0

When using the apply function to apply a function to a DataFrame column, it ends up mixing the column's rows ordering.

A command like this:
{code:java}
def example_func(df_col):
  return df_col ** 2 

df['col_to_apply_function'] = df.apply(lambda row: example_func(row['col_to_apply_function']), axis=1) {code}
A workaround is to assign the results to a new column instead of the same one, but if the old column is dropped, the same error is produced.

Setting one column as index also didn't work.