av9ash/GitBugs huggingface dataset:
Bug: Cancel finalizing the shuffle merge if the stage is cancelled while waiting until shuffle merge finalize wait time.
Status: Open | Priority: Major | Resolution: 
Created: 26/May/21 18:40
Affects Version/s: 3.1.0

Currently we are not handling the case of stage cancellation and if the stage is cancelled while the stage is about to get merge finalized.

Stage completion will wait for shuffle merge finalize wait time so that all the shuffle merge requests are drained but during this time the stage can be cancelled, if this happens then we need to let all the shuffle services to remove the merged shuffle data for the corresponding shuffle ID.
Bug: Introduce ArrayOps, MapOps and StructOps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/21 01:38
Affects Version/s: 3.2.0

StructType, ArrayType, and MapType are not accepted by DataTypeOps now.

We should handle these complex types.

Please note that arithmetic operations might be applied to these complex types, for example, ps.Series([[1,2,3]]) + ps.Series([[4,5,6]]) should work the same as pd.Series([[1,2,3]]) + pd.Series([[4,5,6]]).
Bug: Optional source codes uploading in k8s
Status: Open | Priority: Major | Resolution: 
Created: 26/May/21 11:03
Affects Version/s: 3.1.1

It seems there is currently a hard requirement on uploading files in k8s mode via {{spark.kubernetes.file.upload.path config option that actually doesn't make sense if your docker image already contains all of the necessary files.}}

I would propose a local:// extension to spark.kubernetes.file.upload.path, that will point to the local folder containing files without their actual upload.
Bug: MemoryConsumer reservations that trigger a partial self-spill can fail even if memory is available
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/21 18:58
Affects Version/s: 2.4.8

When a memory reservation triggers a self-spill, ExecutionMemoryPool#releaseMemory() will immediately notify waiting tasks that memory has been freed. If there are any waiting tasks with less than 1/2N of the memory pool, they may acquire the newly-freed memory before the current task has a chance to do so. This will cause the original memory reservation to fail. If the initial spill did not release all available memory, the reservation could have been satisfied by asking it to spill again.

For example, the following test fails when added to MemoryManagerSuite:

{code:scala}
test("SPARK-35486: memory freed by self-spilling is taken by another task") {
    val memoryManager = createMemoryManager(1000L)
    val t1MemManager = new TaskMemoryManager(memoryManager, 1)
    val t2MemManager = new TaskMemoryManager(memoryManager, 2)
    val c1 = new TestPartialSpillingMemoryConsumer(t1MemManager)
    val c2 = new TestMemoryConsumer(t2MemManager)
    val futureTimeout: Duration = 20.seconds

    // t1 acquires 1000 bytes. This should succeed immediately.
    val t1Result1 = Future { c1.acquireMemory(1000L) }
    assert(ThreadUtils.awaitResult(t1Result1, futureTimeout) === 1000L)
    assert(c1.getUsed() === 1000L)

    // t2 attempts to acquire 500 bytes. This should block since there is no memory available.
    val t2Result1 = Future { c2.acquireMemory(500L) }
    Thread.sleep(300)
    assert(!t2Result1.isCompleted)
    assert(c2.getUsed() === 0L)

    // t1 attempts to acquire 500 bytes, causing its existing reservation to spill partially. t2 is
    // first in line for the freed memory.
    //
    // SPARK-35486: This currently causes the reservation to fail. Instead, t1 should try again,
    // causing the rest of the reservation to spill.
    val t1Result2 = Future { c1.acquireMemory(500L) }

    // The spill should release enough memory for both t1's and t2's reservations to be satisfied.
    assert(ThreadUtils.awaitResult(t2Result1, futureTimeout) === 500L)
    // SPARK-35486: This assertion fails: 0L != 500L.
    // assert(ThreadUtils.awaitResult(t1Result2, futureTimeout) === 500L)
  }
{code}


{code:java}
/**
 * A TestMemoryConsumer which, when asked to spill, releases only enough memory to satisfy the
 * request rather than releasing all its memory.
 */
public class TestPartialSpillingMemoryConsumer extends TestMemoryConsumer {
  public TestPartialSpillingMemoryConsumer(TaskMemoryManager memoryManager, MemoryMode mode) {
    super(memoryManager, mode);
  }
  public TestPartialSpillingMemoryConsumer(TaskMemoryManager memoryManager) {
    super(memoryManager);
  }

  @Override
  public long spill(long size, MemoryConsumer trigger) throws IOException {
    long used = getUsed();
    long released = Math.min(used, size);
    free(released);
    return released;
  }
}
{code}
Bug: structured streaming multiple queries can not be execute concurrently 
Status: Open | Priority: Major | Resolution: 
Created: 21/May/21 18:26
Affects Version/s: 3.1.1

Hi There,

 

I have similar problem as described in an earlier issue reported issue of SPARK-23766. The issue was closed. though would like to know if multiple queries are supported in structured streaming and how to manage them.
Bug: Spark computes all rows during count() on a parquet file
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/May/21 08:44
Affects Version/s: 3.0.0

We expect spark uses parquet metadata to fetch the rows count of a parquet file. But when we execute the following code 
{code:java}
import org.apache.spark.SparkConf
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.catalyst.encoders.{ExpressionEncoder, RowEncoder}

object Test extends App {
  val sparkConf = new SparkConf()
    .setAppName("test-app")
    .setMaster("local[1]")

  val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()
  import sparkSession.implicits._

  val filePath = "./tempFile.parquet"
  (1 to 1000).toDF("c1")
    .repartition(10)
    .write
    .mode("overwrite")
    .parquet(filePath)

  val df = sparkSession.read.parquet(filePath)

  var rowsInHeavyComputation = 0
  def heavyComputation(row: Row): Row = {
    rowsInHeavyComputation += 1
    println(s"rowsInHeavyComputation = $rowsInHeavyComputation")
    Thread.sleep(50)
    row
  }

  implicit val encoder: ExpressionEncoder[Row] = RowEncoder(df.schema)
  val cnt = df
    .map(row => heavyComputation(row)) // map operation cannot change number of rows 
    .count()
  println(s"counting done, cnt=$cnt")
}
{code}
we see 
{code:java}
rowsInHeavyComputation = 1
rowsInHeavyComputation = 2
...
rowsInHeavyComputation = 999
rowsInHeavyComputation = 1000
counting done, cnt=1000
{code}
 *Expected result* - spark does not perform heavyComputation at all.

 

P.S. In our real application we:
 - transform data from parquet files
 - return some examples (50 rows and spark does heavyComputation only for 50 rows)
 - return rows count of the whole DataFrame and here spark for some reason computes the whole DataFrame despite the fact there are only map operations and initial rows count can be gotten from parquet meta
Bug: Define UDTs in schemas using string format
Status: Open | Priority: Major | Resolution: 
Created: 26/May/21 02:16
Affects Version/s: 3.1.1

In PySpark where UDTs are public in 3.1.1 for example, you can define a schema using UDTs in the format:

schema = StructType([StructField("Stuff", MyUDT())])

but the format

schema = "Stuff MyUDT"

does not work.

UDTs are officially being made public again in 3.2.0 for Scala, so this issue is pretty important now.
Bug: List Python 3.8 installed libraries in build_and_test workflow
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/May/21 16:43
Affects Version/s: 3.2.0

In the build_and_test workflow, tests are ran against both Python 3.6 and Python 3.8. However, only libraries installed in Python 3.6 are listed. We should list Python 3.8's installed libraries as well.
Bug: Should not extract common expressions from value expressions when elseValue is empty in CaseWhen
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/21 19:37
Affects Version/s: 3.1.1

[https://github.com/apache/spark/pull/30245]  added support for creating subexpressions that are present in all branches of conditional statements. However, for a statement to be in "all branches" of a CaseWhen statement, it must also be in the elseValue. This can lead to a subexpression to be created and run for branches of a conditional that don't pass. This can cause issues especially with a UDF in a branch that gets executed assuming the condition is true. For example:
{code:java}
val col = when($"id" < 0, myUdf($"id"))
spark.range(1).select(when(col > 0, col)).show()
{code}
myUdf($"id") gets extracted as a subexpression and executed even though both conditions don't pass and it should never be executed.
Bug: Spark Executor metrics are not produced/showed
Status: Open | Priority: Major | Resolution: 
Created: 24/May/21 13:30
Affects Version/s: 3.0.1

Recently we have enabled prometheusServlet configuration in order to have spark master, worker, driver and executor metrics.

We can see and using spark master, worker and driver executors but can't see spark executor metrics.

We are running spark streaming standalone cluster in version 3.0.1 over physical servers.

 
 We have taken one of our jobs and added the following parameters to the job configuration, but couldn't see executer metrics by curling both driver and executor workers of this job:
  
 These are the parameters:
 --conf spark.ui.prometheus.enabled=true \
 --conf spark.executor.processTreeMetrics.enabled=true
  
 Curl commands:
 [00764f](root@sparktest-40005-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
 [00764f](root@sparktest-40005-prod-chidc2:~)#
 Driver of this job - sparktest-40004:
 [e35005](root@sparktest-40004-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
 [e35005](root@sparktest-40004-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
  
 Our UI port is on 4050
  
  I understood that the executor Prometheus endpoint is still experimental which may explain the inconsistent behaviour we see but is there a plan to fix it?
  
 Are there any known issues regarding this?
Bug: Timestamp casting performance issue when invoked with timezone
Status: In Progress | Priority: Major | Resolution: 
Created: 23/May/21 15:28
Affects Version/s: 2.4.7

In Spark SQL 2.4, when converting a datetime string column to timestamp with cast or to_timestamp, we have noticed a major performance issue when the source string contains timezone information (for example 2021-05-24T00:00:00+02:00)

This simple benchmark illustrates the difference:
{noformat}
OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz

Timestamp Conversion: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------
no tz                 1368 / 1379       3,1       326,1       1,0X
with UTC tz           5940 / 5947       0,7       1416,2      0,2X
with hours tz         5940 / 5962       0,7       1416,2      0,2X
{noformat}
With the provided patch, the benchmark results are:
{noformat}
OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz

Timestamp Conversion: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
no tz                 1383 / 1392       3,0       329,7       1,0X
with UTC tz           1550 / 1587       2,7       369,6       0,9X
with hours tz         1570 / 1574       2,7       374,2       0,9X
{noformat}
 This does not occur on the 3.x branches
Bug: Unused subexpressions leftover in WholeStageCodegen subexpression elimination
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/21 14:13
Affects Version/s: 3.1.1

Trying to understand and debug the performance of some of our jobs, I started digging into what the whole stage codegen code was doing. We use a lot of case when statements, and I found that there were a lot of unused sub expressions that were left over after the subexpression elimination, and it gets worse the more expressions you have chained. The simple example:
{code:java}
import org.apache.spark.sql.functions._
import spark.implicits._
val myUdf = udf((s: String) => {
  println("In UDF")
  s.toUpperCase
})
spark.range(5).select(when(length(myUdf($"id")) > 0, length(myUdf($"id")))).show()
{code}
Running the code, you'll see "In UDF" printed out 10 times. And if you change both to log(length(myUdf($"id")), "In UDF" will print out 20 times (one more for a cast from int to double and one more for the actual log calculation I think).

In the codegen for this (without the log), there are these initial subexpressions:
{code:java}
/* 076 */     UTF8String project_subExprValue_0 = project_subExpr_0(project_expr_0_0);
/* 077 */     int project_subExprValue_1 = project_subExpr_1(project_expr_0_0);
/* 078 */     UTF8String project_subExprValue_2 = project_subExpr_2(project_expr_0_0);
{code}
project_subExprValue_0 and project_subExprValue_2 are never actually used, so it's properly resolving the two expressions and sharing the result of project_subExprValue_1, but it's not removing the other sub expression calls it seems like.
Bug: Check the EXCEPTION rebase mode for Avro/Parquet
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/21 15:56
Affects Version/s: 3.2.0

Add tests to check the SparkUpgradeException exception in the EXCEPTION rebase node for Avro and Parquet datasource. Currently, the mode is checked implicitly, and not for all data types columns.
Bug: Format PartitionFilters IN strings in scan nodes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/21 01:32
Affects Version/s: 3.2.0

This ticket proposes to format strings correctly for `PushedFilters`. For example, `explain()` for a query below prints `v in (array('a'))` as `PushedFilters: [In(v, [WrappedArray(a)])]`;
{code}
scala> sql("create table t (v array<string>) using parquet")
scala> sql("select * from t where v in (array('a'), null)").explain()
== Physical Plan ==
*(1) Filter v#4 IN ([a],null)
+- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN ([a],null)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t], PartitionFilters: [], PushedFilters: [In(v, [WrappedArray(a),null])], ReadSchema: struct<v:array<string>>
{code}
This PR makes `explain()` print it as `PushedFilters: [In(v, [[a]])]`; 
{code}
scala> sql("select * from t where v in (array('a'), null)").explain()
== Physical Plan ==
*(1) Filter v#4 IN ([a],null)
+- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN ([a],null)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>
{code}
Bug: Renaming the existing Koalas related codes.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/21 02:46
Affects Version/s: 3.2.0

We should renaming the several Koalas-related codes in the pandas APIs on Spark.
 * kdf -> psdf
 * kser -> psser
 * kidx -> psidx
 * kmidx -> psmidx
 * sdf.to_koalas() -> sdf.to_pandas_on_spark()
Bug: Separate arithmetic operations into data type based structures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/21 16:48
Affects Version/s: 3.2.0

For arithmetic operations:
{code:java}
__add__, __sub__, __mul__, __truediv__, __floordiv__, __pow__, __mod__,
__radd__, __rsub__, __rmul__, __rtruediv__, __rfloordiv__, __rpow__,__rmod__
{code}
we would like to separate them into data-type-based structures.

The existing behaviors of each arithmetic operation should be preserved.
Bug: Improve createTable logic if table exists
Status: In Progress | Priority: Major | Resolution: 
Created: 19/May/21 11:48
Affects Version/s: 3.0.0

It should return directly if table exists and ignoreIfExists=true

 

current logic：
{code:java}
requireDbExists(db)
if (tableExists(newTableDefinition.identifier)) {
  if (!ignoreIfExists) {
    throw new TableAlreadyExistsException(db = db, table = table)
  }
} else if (validateLocation) {
  validateTableLocation(newTableDefinition)
}
externalCatalog.createTable(newTableDefinition, ignoreIfExists)
{code}
Bug: Remove redundant ProjectExec from streaming queries with V2Relation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/21 08:13
Affects Version/s: 3.2.0

Streaming queries with V2Relation can have redundant ProjectExec in it's physical plan.
You can easily reproduce with the following code.

{code}
import org.apache.spark.sql.streaming.Trigger

val query = spark.
  readStream.
  format("rate").
  option("rowsPerSecond", 1000).
  option("rampUpTime", "10s").
  load().
  selectExpr("timestamp", "100",  "value").  
  writeStream.
  format("console").
  trigger(Trigger.ProcessingTime("5 seconds")).
  // trigger(Trigger.Continuous("5 seconds")). // You can reproduce with continuous processing too.
  outputMode("append").
  start()
{code}
Bug: Analyzer should set progagateNull to false for magic function invocation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/21 19:44
Affects Version/s: 3.2.0

For both {{Invoke}} and {{StaticInvoke}} used by magic method of {{ScalarFunction}}, we should set {{propgateNull}} to false, so that null values will be passed to the UDF for evaluation, instead of bypassing that and directly return null.
Bug: Expose TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions via Scala, Python and R APIs
Status: Open | Priority: Major | Resolution: 
Created: 18/May/21 08:36
Affects Version/s: 3.2.0

The PR https://github.com/apache/spark/pull/28534 added new functions TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS but the functions are available to users only via SQL. To make other APIs (Scala/Java, PySpark and R) as powerful as SQL, need to implement the functions in the APIs too.
Bug: Sort elements generated by collect_set in SQLQueryTestSuite 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/21 05:10
Affects Version/s: 3.1.2

To pass {{subquery/scalar-subquery/scalar-subquery-select.sql}} ({{SQLQueryTestSuite}}) in Scala v2.13, this ticket proposes to change the aggregate expr of a test query in the file from {{collect_set(...)}} to {{sort_array(collect_set(...))}} because {{collect_set}} depends on the {{mutable.HashSet}} implementation and elements in the set are printed in a different order in Scala v2.12/v2.13.
Bug: Insert data with char/varchar datatype will fail when data length exceed length limitation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/21 01:23
Affects Version/s: 3.1.1

In Spark3.1.1 have support Char/Varchar type, but when insert data with char/varchar datatype will fail when data length exceed length limitation even when spark.sql.legacy.charVarcharAsString is true.

reproduce:

create table chartb01(a char(3));

insert into chartb01 select 'aaaaa';
Bug: Fix a bug in groupBy of year-month/day-time intervals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/21 09:59
Affects Version/s: 3.2.0

I found a bug in groupBy of year-month/day-time intervals;
{code}
scala> Seq(java.time.Duration.ofDays(1)).toDF("a").groupBy("a").count().show()
scala.MatchError: DayTimeIntervalType (of class org.apache.spark.sql.types.DayTimeIntervalType$)
  at org.apache.spark.sql.execution.aggregate.HashMapGenerator.genComputeHash(HashMapGenerator.scala:159)
  at org.apache.spark.sql.execution.aggregate.HashMapGenerator.$anonfun$generateHashFunction$1(HashMapGenerator.scala:102)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at scala.collection.TraversableLike.map(TraversableLike.scala:238)
  at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
  at scala.collection.immutable.List.map(List.scala:298)
  at org.apache.spark.sql.execution.aggregate.HashMapGenerator.genHashForKeys$1(HashMapGenerator.scala:99)
  at org.apache.spark.sql.execution.aggregate.HashMapGenerator.generateHashFunction(HashMapGenerator.scala:111)
{code}
Bug: Replace SessionState.start with SessionState.setCurrentSessionState
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/21 14:11
Affects Version/s: 3.2.0

To avoid SessionState.createSessionDirs creating too many directories:

https://user-images.githubusercontent.com/5399861/116766834-28ea7080-aa5f-11eb-85ff-07bcaee444e5.png
Bug: Submitting Applications documentation has outdated information about K8s client mode support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/May/21 12:02
Affects Version/s: 3.1.1

[Submitting Applications doc|https://spark.apache.org/docs/latest/submitting-applications.html#master-urls] has outdated information about K8s client mode support.
 It still says "Client mode is currently unsupported and will be supported in future releases".
!image-2021-05-14-15-10-55-987.png!

Whereas it's already supported and [Running Spark on Kubernetes doc|https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode] says that it's supported started from 2.4.0 and has all needed information.
!image-2021-05-14-15-11-11-621.png!
Bug: Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/21 11:56
Affects Version/s: 3.2.0

As discussed in [https://github.com/apache/spark/pull/32253,] Spark cannot distribute `com.github.fommil.netlib:all` due to licensing reasons. This limits the ability of Spark to take advantage out of the box of native libraries that do provide hardware acceleration (SIMD and GPUs).

With `dev.ludovic.netlib:2.0.0`, it is possible to take advantage of native libraries, but without the licensing limitations, since it doesn't link nor distribute any GPL or LGPL libraries.

That would allow any other of Spark to install a native library like OpenBLAS or Intel MKL and take advantage transparently of these libraries.
Bug: Move kubernetes-client.version to root pom file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/21 04:32
Affects Version/s: 3.2.0

Currently, Apache Spark has two K8s version variables in two `pom.xml`s. We had better unify it.
- kubernetes.client.version (kubernetes/core module
- kubernetes-client.version (kubernetes/integration-test module)
Bug: Improve s3a magic committer support by inferring missing configs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/21 05:37
Affects Version/s: 3.2.0

{code}
Exception in thread "main" org.apache.hadoop.fs.s3a.commit.PathCommitException: `s3a://my-spark-bucket`: Filesystem does not have support for 'magic' committer enabled in configuration option fs.s3a.committer.magic.enabled
	at org.apache.hadoop.fs.s3a.commit.CommitUtils.verifyIsMagicCommitFS(CommitUtils.java:74)
	at org.apache.hadoop.fs.s3a.commit.CommitUtils.getS3AFileSystem(CommitUtils.java:109)
{code}