av9ash/GitBugs huggingface dataset:
Bug: Do not wrap exceptions thrown in FileFormatWriter.write with SparkException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Sep/22 10:42
Affects Version/s: 3.4.0

Exceptions thrown in FileFormatWriter.write are wrapped with SparkException("Job aborted.").

This wrapping provides little extra information, but generates a long stacktrace, which hinders debugging when error happens.
Bug: Migrate JSON type check failures onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/22 16:48
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in JSON expressions:
1. JsonTuple (2): https://github.com/apache/spark/blob/9b885ae3ba70cd97489e8768a335c9b9c10e9d87/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L395-L399
2. JsonToStructs (2): https://github.com/apache/spark/blob/9b885ae3ba70cd97489e8768a335c9b9c10e9d87/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L570-L574
3. StructsToJson (4): https://github.com/apache/spark/blob/9b885ae3ba70cd97489e8768a335c9b9c10e9d87/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L725-L743
4. SchemaOfJson (1): https://github.com/apache/spark/blob/9b885ae3ba70cd97489e8768a335c9b9c10e9d87/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L806
Bug: Upgrade Apache Kafka from 3.2.1 to 3.2.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/22 15:18
Affects Version/s: 3.4.0

[Memory Allocation with Excessive Size Value SNYK-JAVA-ORGAPACHEKAFKA-3027430 |https://security.snyk.io/vuln/SNYK-JAVA-ORGAPACHEKAFKA-3027430]
Bug: Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Sep/22 10:07
Affects Version/s: 3.4.0

The 'Part 1' and 'Part 2' could run in parallel
{code:java}
  /**
   * The implementation for these joins:
   *
   *   LeftOuter with BuildLeft
   *   RightOuter with BuildRight
   *   FullOuter
   */
  private def defaultJoin(relation: Broadcast[Array[InternalRow]]): RDD[InternalRow] = {
    val streamRdd = streamed.execute()

    // Part 1
    val matchedBroadcastRows = getMatchedBroadcastRowsBitSet(streamRdd, relation)
    val notMatchedBroadcastRows: Seq[InternalRow] = {
      val nulls = new GenericInternalRow(streamed.output.size)
      val buf: CompactBuffer[InternalRow] = new CompactBuffer()
      val joinedRow = new JoinedRow
      joinedRow.withLeft(nulls)
      var i = 0
      val buildRows = relation.value
      while (i < buildRows.length) {
        if (!matchedBroadcastRows.get(i)) {
          buf += joinedRow.withRight(buildRows(i)).copy()
        }
        i += 1
      }
      buf
    }

    // Part 2
    val matchedStreamRows = streamRdd.mapPartitionsInternal { streamedIter =>
      val buildRows = relation.value
      val joinedRow = new JoinedRow
      val nulls = new GenericInternalRow(broadcast.output.size)

      streamedIter.flatMap { streamedRow =>
        var i = 0
        var foundMatch = false
        val matchedRows = new CompactBuffer[InternalRow]

        while (i < buildRows.length) {
          if (boundCondition(joinedRow(streamedRow, buildRows(i)))) {
            matchedRows += joinedRow.copy()
            foundMatch = true
          }
          i += 1
        }

        if (!foundMatch && joinType == FullOuter) {
          matchedRows += joinedRow(streamedRow, nulls).copy()
        }
        matchedRows.iterator
      }
    }

    // Union
    sparkContext.union(
      matchedStreamRows,
      sparkContext.makeRDD(notMatchedBroadcastRows)
    )
  }{code}
Bug: local mode with resource scheduling can hang
Status: Open | Priority: Major | Resolution: 
Created: 21/Sep/22 22:35
Affects Version/s: 3.1.0

If you try to run spark in local mode and request custom resources like GPU's, Spark will hang.  Resource scheduling isn't supported in local mode so just removing the request for resources fixes the issue, but its really confusing to users since it just hangs.

 

ie to reproduce:

spark-sql --conf spark.executor.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=1

Run:

select 1

result: hangs

To fix run:

spark-sql 

 

spark-sql> select 1;
1
Time taken: 2.853 seconds, Fetched 1 row(s)

 

It would be nice if we just fail to start or threw an exception when using those options in local mode
Bug: Remove min heap setting in Kubernetes Dockerfile entrypoint
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Sep/22 11:59
Affects Version/s: 3.2.2

The entrypoint script for the Kubernetes Dockerfile sets the Java min heap setting (-Xms) to be the same as the max setting (-Xmx) for the executor process. This prevents the JVM from shrinking the heap and can lead to excessive memory usage in some scenarios. Removing the min heap setting is consistent with YARN executor startup.
Bug: Allow job status tracking with jobGroupId
Status: Open | Priority: Major | Resolution: 
Created: 16/Sep/22 18:45
Affects Version/s: 3.3.0

Spark let's us group jobs together by setting a job group id. This is useful to check the job group in the web UI. For example

{{spark.sparkContext().setJobGroup("mygroup_id")}}

We have a use-case where we would like to have a long running Spark application and have jobs submitted to it. We would like to programmatically check the status of the jobs created by this group id. For example, [SQLStatusStore|#L41]] has `executionList()` which returns a map of jobs to the status. There is no way to filter this based on jobGroupId. 

This Jira is to add ability to get fine grained job statues by jobGroupId.
Bug: Avoid creating directory failures
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Sep/22 04:52
Affects Version/s: 3.4.0

{noformat}
java.nio.file.NoSuchFileException: /hadoop/3/yarn/local/usercache/b_carmel/appcache/application_1654776504115_37917/blockmgr-e18b484f-8c49-4c7d-b649-710439b0e4c3/3c
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:123)
	at org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:146)
	at org.apache.spark.storage.DiskStore.contains(DiskStore.scala:147)
	at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:853)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$4(TorrentBroadcast.scala:253)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$2(TorrentBroadcast.scala:250)
	at org.apache.spark.util.KeyLock.withLock(KeyLock.scala:64)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$1(TorrentBroadcast.scala:245)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1383)
	at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:245)
	at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:109)
	at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:86)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:132)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:487)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1417)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:490)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{noformat}
Bug: Abort result stage directly when it failed caused by FetchFailed
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Sep/22 11:41
Affects Version/s: 3.0.0

Here's a very serious bug：

When result stage failed caused by FetchFailedException,  the previous condition to determine whether result stage retries are allowed is {color:#ff0000}numMissingPartitions < resultStage.numTasks{color}. 

 

If this condition holds on retry, but the other tasks at the current result stage are not killed, when result stage was resubmit, it would got wrong partitions to recalculation.
{code:java}
// DAGScheduler#submitMissingTasks
 
// Figure out the indexes of partition ids to compute.
val partitionsToCompute: Seq[Int] = stage.findMissingPartitions() {code}
It is possible that the number of partitions to be recalculated is smaller than the actual number of partitions at result stage
Bug: Migrate cast type check failures onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/22 09:28
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the CAST expression:
typeCheckFailureMessage (6): https://github.com/apache/spark/blob/f8d51b9940b5f1f7c1f37693b10931cbec0a4741/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L428-L455
Bug: Support applyInPandasWithState in batch query
Status: Open | Priority: Major | Resolution: 
Created: 15/Sep/22 07:29
Affects Version/s: 3.4.0

In Scala/Java, flatMapGroupsWithState API uses different physical plan for batch query, which applies some wrap-up with user function to effectively match up with user function in MapGroupsExec.

This seems to be non-trivial to deal with in PySpark, or at least seems to require major work.  This ticket tracks the effort of supporting the API in batch query.
Bug: Use YuniKorn v1.1+
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/22 18:25
Affects Version/s: 3.3.1

YuniKorn 1.1.0 starts to support multi-arch officially.

[https://yunikorn.apache.org/release-announce/1.1.0]
{code:java}
$ docker inspect apache/yunikorn:scheduler-1.0.0 | grep Architecture
        "Architecture": "amd64",
$ docker inspect apache/yunikorn:scheduler-1.1.0 | grep Architecture
        "Architecture": "arm64", {code}
Bug: Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/22 12:53
Affects Version/s: 3.4.0

During the work of flatMapGroupsWithState in PySpark, we figured out that we are unable to reuse PythonArrowInput and PythonArrowOutput, as PythonArrowInput and PythonArrowOutput are too specific to the strict input data (row) and output data.

To reuse the implementations we should make these traits more general to handle more generic type of data.
Bug: Increase default initialNumPartitions to 10
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/22 19:05
Affects Version/s: 3.4.0

It's actually an follow-up from SPARK-40211

The previous default value for initialNumPartitions is 1, which is way too small. Change to 10 might be a reasonable value to achieve a middle-ground trade-off, and will be beneficial in most cases (unless partition num is too small, but in that case either initialNumPartitions and scaleUpFactor won't have significant effect)
Bug: Bug in Canonicalization of expressions like Add & Multiply i.e Commutative Operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/22 22:18
Affects Version/s: 3.3.0

In the canonicalization code which is now in two stages, canonicalization involving Commutative operators is broken, if they are subexpressions of certain type of expressions which override precanonicalize, for example BinaryComparison 

Consider following expression:

a + b > 10

         GT

            |

a + b          10

The BinaryComparison operator in the precanonicalize, first precanonicalizes  children & then   may swap operands based on left /right hashCode inequality..

lets say  Add(a + b) .hashCode is >  10.hashCode as a result GT is converted to LT

But If the same tree is created 

           GT

            |

 b + a      10

The hashCode of Add(b, a) is not same as Add(a, b) , thus it is possible that for this tree

 Add(b + a) .hashCode is <  10.hashCode  in which case GT remains as is.

Thus to similar trees result in different canonicalization , one having GT other having LT 

 

The problem occurs because  for commutative expressions the canonicalization normalizes the expression with consistent hashCode which is not the case with precanonicalize as the hashCode of commutative expression 's precanonicalize and post canonicalize are different.

 

 

The test 
{quote}test("bug X")
Unknown macro: \{     val tr1 = LocalRelation('c.int, 'b.string, 'a.int)    val y = tr1.where('a.attr + 'c.attr > 10).analyze    val fullCond = y.asInstanceOf[Filter].condition.clone()   val addExpr = (fullCond match Unknown macro}
).clone().asInstanceOf[Add]
val canonicalizedFullCond = fullCond.canonicalized

// swap the operands of add
val newAddExpr = Add(addExpr.right, addExpr.left)

// build a new condition which is same as the previous one, but with operands of //Add reversed 

val builtCondnCanonicalized = GreaterThan(newAddExpr, Literal(10)).canonicalized

assertEquals(canonicalizedFullCond, builtCondnCanonicalized)
}
{quote}
This test fails.

The fix which I propose is that for the commutative expressions, the precanonicalize should be overridden and  Canonicalize.reorderCommutativeOperators be invoked on the expression instead of at place of canonicalize. effectively for commutative operands ( add, or , multiply , and etc) canonicalize and precanonicalize should be same.

PR:

[https://github.com/apache/spark/pull/37824]

 

 

I am also trying a better fix, where by the idea is that for commutative expressions the murmur hashCode are caluculated using unorderedHash so that it is order  independent ( i.e symmetric).

The above approach works fine , but in case of Least & Greatest, the Product's element is  a Seq,  and that messes with consistency of hashCode.
Bug: Pass error message parameters to exceptions as a map
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/22 16:58
Affects Version/s: 3.4.0

Modify Spark exception to pass message parameters as a map not an array. At the moment, we still depend on the order of parameters in error-classes.json , so, we can change the text of error messages but not the order of parameters. For example, pass Map[String, String]  instead of Array[String]  in exceptions like:

{code:scala}
private[spark] class SparkRuntimeException(
    errorClass: String,
    errorSubClass: Option[String] = None,
    messageParameters: Array[String]
    ...)
{code}
It should be replaced by:

{code:scala}
    new SparkRuntimeException(
      errorClass = "UNSUPPORTED_FEATURE",
      errorSubClass = "LITERAL_TYPE",
      messageParameters = Map(
        "value" -> v.toString,
        "type" -> v.getClass.toString))
{code}
Bug: limit(x,y) + 子查询 出现数据丢失和乱序问题
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 13/Sep/22 03:39
Affects Version/s: 2.4.5

select * 

from(

select * from

table

limit 10，20

)

结果只有10条  并且不是第11条到第20条  顺序也不对

 

select * from

table

limit 10，20

结果是20条，顺序是11到第30条

select * 

from(

select * from

table

order by id

limit 10，20

)

结果是20条，且顺序也是11到30条
Bug: Implement `SeriesGroupBy.describe`.
Status: Open | Priority: Major | Resolution: 
Created: 05/Sep/22 04:55
Affects Version/s: 3.4.0

We should implement `SeriesGroupBy.describe` for increasing pandas API coverage.

pandas docs: https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.DataFrameGroupBy.describe.html
Bug: StructType level metadata 
Status: Open | Priority: Major | Resolution: 
Created: 10/Sep/22 11:09
Affects Version/s: 3.3.1

Please support StructType level metadata like StructField does. It allows data engineers provide better documentation of data schemas used in pipelines and store this metadata it in catalog. This feature also will simplify table level comment propagation from RDBMS to Spark
Bug: Decimals can't upcast as integral types if the cast can overflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/22 07:12
Affects Version/s: 3.3.1

In Spark SQL, the method "canUpCast" returns true iff we can safely up-cast the `from` type to `to` type without any truncating or precision lose or possible runtime failures.

Meanwhile, DecimalType(10, 0) is considered as "canUpCast" to Integer type. This is wrong, since casting 9000000000BD as Integer type will overflow.

As a result:
 * The optimizer rule SimplifyCasts replies on the method "canUpCast" and it will mistakenly convert "cast(cast(9000000000BD as int) as long)" as "cast(9000000000BD as long)"
 * The STRICT store assignment policy relies on this method too. With the policy enabled, inserting 9000000000BD into integer columns will pass compiling time check and insert an unexpected value 410065408.
Bug: SQL configuration spark.sql.mapKeyDedupPolicy not always applied
Status: Open | Priority: Major | Resolution: 
Created: 08/Sep/22 06:57
Affects Version/s: 3.1.2

I have set spark.sql.mapKeyDedupPolicy to LAST_WIN.

However, I had still one failure where I got
{quote}Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 1201.0 failed 4 times, most recent failure: Lost task 7.3 in stage 1201.0 (TID 1011313) (ip-10-1-34-47.eu-west-1.compute.internal executor 228): java.lang.RuntimeException: Duplicate map key domain was found, please check the input data. If you want to remove the duplicated keys, you can set spark.sql.mapKeyDedupPolicy to LAST_WIN so that the key inserted at last takes precedence.
{quote}
We are confident we set the right configuration in SparkConf (we can find it on the Spark UI -> Environment).

It is our impression this configuration is not propagated reliably to the executors.
Bug: Allow customize maxBroadcastTableBytes and maxBroadcastRows
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Sep/22 12:41
Affects Version/s: 3.4.0

Recently, we encountered some driver OOM problems. Some large tables were compressed using Snappy and then broadcast join was performed, but the actual data volume was too large, which resulted in driver OOM.

The values of maxBroadcastTableBytes and maxBroadcastRows are hardcoded, 8GB and 512000000 respectively. Maybe we can allow customization of these values, configure smaller values according to different scenarios, and prohibit broadcast joins for some large tables to avoid driver OOM.
Bug: Support eliminate dynamic partition for v1 writes
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Sep/22 09:47
Affects Version/s: 3.4.0

v1 writes will add an extra sort for dynamic columns, e.g.
{code:java}
INSERT INTO TABLE t1 PARTITION(p)
SELECT c1, c2, 'a' as p FROM t2 {code}
if the dynamic columns are foldable, we can optimize it to:
{code:java}
INSERT INTO TABLE t1 PARTITION(p='a')
SELECT c1, c2 FROM t2 {code}
Bug: try_avg() should throw the exceptions from its child
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/22 20:29
Affects Version/s: 3.4.0

milar to [https://github.com/apache/spark/pull/37486] and [https://github.com/apache/spark/pull/37663,]  the errors from try_avg()'s child should be shown instead of ignored.
Bug: Non-deterministic hashCode() calculations for ArrayBasedMapData on equal objects
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/22 12:16
Affects Version/s: 3.2.2

There is no explicit `hashCode()` function override for the `ArrayBasedMapData` LogicalPlan. As a result, the `hashCode()` computed for `ArrayBasedMapData` can be different for two equal objects (objects with equal keys and values).

This error is non-deterministic and hard to reproduce, as we don't control the default `hashCode()` function.

We should override the `hashCode` function so that it works exactly as we expect. We should also have an explicit `equals()` function for consistency with how `Literals` check for equality of `ArrayBasedMapData`.
Bug: Add function aliases: len, datepart, dateadd, date_diff and curdate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Sep/22 14:58
Affects Version/s: 3.4.0

The functions len, datepart, dateadd, date_diff and curdate exist in other systems, and Spark SQL has similar functions. So, adding such aliases will make the migration to Spark SQL easier.
Bug: Implement `DataFrameGroupBy.cov`.
Status: Open | Priority: Major | Resolution: 
Created: 05/Sep/22 04:53
Affects Version/s: 3.4.0

We should implement `DataFrameGroupBy.cov` for increasing pandas API coverage.

pandas docs: https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.DataFrameGroupBy.cov.html
Bug: Implement `DataFrameGroupBy.corr`.
Status: Open | Priority: Major | Resolution: 
Created: 05/Sep/22 04:52
Affects Version/s: 3.4.0

We should implement `DataFrameGroupBy.corr` for increasing pandas API coverage.

pandas docs: https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.DataFrameGroupBy.corr.html
Bug: Use spark config to configure the parameters of volcano podgroup
Status: In Progress | Priority: Major | Resolution: 
Created: 05/Sep/22 09:47
Affects Version/s: 3.3.0

Now we use volcano as our scheduler， we need  specify  the following configuration options:

 

 
{code:java}
spark.kubernetes.scheduler.name=volcano
spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/path/to/podgroup-template.yaml

spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep

spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep {code}
 

 

we should use configMap to mount the /path/to/podgroup-template.yaml file

 

If we use spark config to specify the parameters of the podgroup, it will be much more convenient,we don't need configmap to mount static files

 

In our scenario, we need to dynamically specify the volcano queue, but it is not convenient to create a static podgroup configuration file to mount
Bug: Improvement to JDBC predicate for queries involving joins
Status: Open | Priority: Major | Resolution: 
Created: 02/Sep/22 18:14
Affects Version/s: 3.2.2

Current behaviour on tables involving joins seems to use a subquery as follows

 

select * from

(

select a, b, c from tbl1

lj tbl2 on tbl1.col1 = tbl2.col1

lj tbl3 on tbl1.col2 = tbl3.col2

)

where predicate = 1

where predicate = 2

where predicate = 3

 

More desirable would be

(

select a, b, c from tbl1 where (predicate = 1, predicate = 2, etc)

lj tbl2 on tbl1.col1 = tbl2.col1

lj tbl3 on tbl1.col2 = tbl3.col2

)

 

to just do the join on the subset of data rather than joining all data then filtering.  Predicate pushdown usually only works on columns that have been indexes.  So even if the data isn't indexed, this would reduce amount of data needing to be moved.  In many cases better to do the join on DB side than pulling everything into Spark.