av9ash/GitBugs huggingface dataset:
Bug: Delay execution hash until after aggregation for semi-join runtime filter.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/22 11:13
Affects Version/s: 2.4.0

Currently, Spark runtime filter supports bloom filter and in subquery filter.
The in subquery filter always execute Murmur3Hash before aggregate the join key.

Because the data size before aggregate will lager than after, we can delay execute Murmur3Hash until after aggregation for semi-join runtime filter and it will reduce the number of calls to Murmur3Hash and improve performance.
Bug: stats and constraints in LogicalRDD may not be in sync with output attributes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/22 02:38
Affects Version/s: 3.4.0

We encountered the case where the output of logical plan and optimized plan were different in LogicalRDD (the difference was exprId for the case), led the situation that stats and constraints are out of sync with output attributes, eventually failed the query.

We should remap stats and constraints based on the output of logical plan, assuming that the output of logical plan and optimized plan are "slightly" different (e.g. exprId) but "semantically" same.
Bug: Upgrade org.scalatestplus:selenium-4-4 to org.scalatestplus:selenium-4-7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 11:43
Affects Version/s: 3.4.0

https://github.com/scalatest/scalatestplus-selenium/releases/tag/release-3.2.14.0-for-selenium-4.7
Bug: Oneof field support and recursive fields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/22 21:29
Affects Version/s: 2.3.0

we should add support for protobuf OneOf fields to Spark-Protobuf. This will involve implementing logic to detect when a protobuf message contains a OneOf field, and to handle it appropriately when using from_protobuf and to_protobuf. 

we should add unit tests to ensure that the implementation of protobuf OneOf field support is correct.

Users can use protobuf OneOf fields with Spark-protobuf, making it more complete and useful for processing protobuf data.
Bug: Make "Rule id not found" error message more actionable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 23:02
Affects Version/s: 3.3.0

It was super confusing to me when adding a new rule that I bumped into the rule id error. We should update the error message to make it more actionable, i.e. explaining to the developers which file to modify.
Bug: Allow Generate with no required child output to host outer references
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/22 16:47
Affects Version/s: 3.4.0

Currently, in CheckAnalysis, Spark disallows Generate to host any outer references when it's required child output is not empty. But when the child output is empty, it can host outer references, which DecorrelateInnerQuery does not handle.

For example,
{code:java}
select * from t, lateral (select explode(array(c1, c2))){code}
This throws an internal error :
{code:java}
Caused by: java.lang.AssertionError: assertion failed: Correlated column is not allowed in Generate explode(array(outer(c1#219), outer(c2#220))), false, [col#221] +- OneRowRelation{code}
 We should support Generate to host outer references when its required child output is empty.
Bug: Iterator returned by GenerateExec isn't interruptible which prevents task from being killed even after cancellation
Status: Open | Priority: Major | Resolution: 
Created: 18/Dec/22 16:10
Affects Version/s: 3.3.1

Iterator returned by GenerateExec isn't interruptible which prevents task from being killed even after cancellation.
{code:java}
val rows = if (requiredChildOutput.nonEmpty) {

  val pruneChildForResult: InternalRow => InternalRow =
    if (child.outputSet == AttributeSet(requiredChildOutput)) {
      identity
    } else {
      UnsafeProjection.create(requiredChildOutput, child.output)
    }

  val joinedRow = new JoinedRow
  iter.flatMap { row =>
    // we should always set the left (required child output)
    joinedRow.withLeft(pruneChildForResult(row))
    val outputRows = boundGenerator.eval(row)
    if (outer && outputRows.isEmpty) {
      joinedRow.withRight(generatorNullRow) :: Nil
    } else {
      outputRows.toIterator.map(joinedRow.withRight)
    }
  } ++ LazyIterator(() => boundGenerator.terminate()).map { row =>
    // we leave the left side as the last element of its child output
    // keep it the same as Hive does
    joinedRow.withRight(row)
  }
} else {
  iter.flatMap { row =>
    val outputRows = boundGenerator.eval(row)
    if (outer && outputRows.isEmpty) {
      Seq(generatorNullRow)
    } else {
      outputRows
    }
  } ++ LazyIterator(() => boundGenerator.terminate())
}

// Convert the rows to unsafe rows.
val proj = UnsafeProjection.create(output, output)
proj.initialize(index)
rows.map { r =>
  numOutputRows += 1
  proj(r)
} {code}
Even the iterator returned in codegen code isn't interruptible. 

The problem with this is that Spark task cannot be cancelled due to stage fail (without interruptThread enabled, which is disabled by default), which wastes executor resources.
Bug: Support partition filter in MSCK REPAIR TABLE statement
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Dec/22 14:20
Affects Version/s: 3.3.1

Since listing all partitions from metastore or directories from filesystem is a heavy operation, we can improve the performance if there are only some specific partitions need to repair.

We would like to expand the *{{MSCK REPAIR TABLE}}* statement to support repair partitions in some specific partition values.
Bug: Disable Coverage in python.pyspark.tests.test_memory_profiler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/22 07:58
Affects Version/s: 3.4.0

https://github.com/apache/spark/actions/runs/3712125552/jobs/6293347848

{code}
======================================================================
FAIL [13.173s]: test_memory_profiler (pyspark.tests.test_memory_profiler.MemoryProfilerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/tests/test_memory_profiler.py", line 56, in test_memory_profiler
    self.assertTrue("plus_one" in fake_out.getvalue())
AssertionError: False is not true

======================================================================
FAIL [3.986s]: test_profile_pandas_function_api (pyspark.tests.test_memory_profiler.MemoryProfilerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/tests/test_memory_profiler.py", line 87, in test_profile_pandas_function_api
    self.assertTrue(f_name in fake_out.getvalue())
AssertionError: False is not true

======================================================================
FAIL [3.722s]: test_profile_pandas_udf (pyspark.tests.test_memory_profiler.MemoryProfilerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/tests/test_memory_profiler.py", line 69, in test_profile_pandas_udf
    self.assertTrue(f_name in fake_out.getvalue())
AssertionError: False is not true

----------------------------------------------------------------------
Ran 3 tests in 20.882s
{code}
Bug: Stages UI page fails to load for proxy in some yarn versions 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/22 09:46
Affects Version/s: 3.3.1

My environment CDH 5.8 , click to enter the spark UI from the yarn interface
when visit the stage URI, it fails to load,  URI is
{code:java}
http://<yarn-url>:8088/proxy/application_1669877165233_0021/stages/stage/?id=0&attempt=0 {code}
!image-2022-12-02-17-53-03-003.png|width=430,height=697!

Server error stack trace:
{code:java}
Caused by: java.lang.NullPointerException
	at org.apache.spark.status.api.v1.StagesResource.$anonfun$doPagination$1(StagesResource.scala:207)
	at org.apache.spark.status.api.v1.BaseAppResource.$anonfun$withUI$1(ApiRootResource.scala:142)
	at org.apache.spark.ui.SparkUI.withSparkUI(SparkUI.scala:147)
	at org.apache.spark.status.api.v1.BaseAppResource.withUI(ApiRootResource.scala:137)
	at org.apache.spark.status.api.v1.BaseAppResource.withUI$(ApiRootResource.scala:135)
	at org.apache.spark.status.api.v1.StagesResource.withUI(StagesResource.scala:31)
	at org.apache.spark.status.api.v1.StagesResource.doPagination(StagesResource.scala:206)
	at org.apache.spark.status.api.v1.StagesResource.$anonfun$taskTable$1(StagesResource.scala:161)
	at org.apache.spark.status.api.v1.BaseAppResource.$anonfun$withUI$1(ApiRootResource.scala:142)
	at org.apache.spark.ui.SparkUI.withSparkUI(SparkUI.scala:147)
	at org.apache.spark.status.api.v1.BaseAppResource.withUI(ApiRootResource.scala:137)
	at org.apache.spark.status.api.v1.BaseAppResource.withUI$(ApiRootResource.scala:135)
	at org.apache.spark.status.api.v1.StagesResource.withUI(StagesResource.scala:31)
	at org.apache.spark.status.api.v1.StagesResource.taskTable(StagesResource.scala:145)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62){code}
 

The issue is similar to, the final phenomenon of the issue is the same, because the parameter encode twice
https://issues.apache.org/jira/browse/SPARK-32467
https://issues.apache.org/jira/browse/SPARK-33611

The two issues solve two scenarios to avoid encode twice:
1. https redirect proxy
2. set reverse proxy enabled (spark.ui.reverseProxy)  in Nginx 

But if encode twice due to other reasons, such as this issue (yarn proxy), it will also fail
Bug: `protoc-jar-maven-plugin` should uniformly use `protoc-jar-maven-plugin.version` as the version 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/22 04:52
Affects Version/s: 3.4.0

[https://github.com/apache/spark/blob/7671bc975f2d88ab929e4982abfe3e166fa72e35/connector/protobuf/pom.xml#L161-L167]

 

!image-2022-12-15-12-53-02-818.png!
Bug: Fix wrong child call in SQLShuffleWriteMetricsReporter.decRecordsWritten()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/22 04:14
Affects Version/s: 3.0.0

In the {{SQLShuffleWriteMetricsReporter.decRecordsWritten}} method, a call to a child accidentally decrements _bytesWritten_ instead of {_}recordsWritten{_}:


{code:java}
  override def decRecordsWritten(v: Long): Unit = {
    metricsReporter.decBytesWritten(v)
    _recordsWritten.set(_recordsWritten.value - v)
  } {code}
One of the situations where {{decRecordsWritten}} is called while reverting shuffle writes from failed/canceled tasks. Due to the mixup in these calls, the _recordsWritten_ metric ends up being _v_ records too high (since it wasn't decremented) and the _bytesWritten_ metric ends up _v_ records too low, causing some failed tasks' write metrics to look like
{code:java}
{"Shuffle Bytes Written":-2109,"Shuffle Write Time":2923270,"Shuffle Records Written":2109} {code}
instead of
{code:java}
{"Shuffle Bytes Written":0,"Shuffle Write Time":2923270,"Shuffle Records Written":0} {code}
I'll submit a fix for this.
Bug: LongToUnsafeRowMap support ignoresDuplicatedKey
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/22 12:11
Affects Version/s: 3.4.0

For left semi and left anti hash join, the duplicated keys of build side have no meaning.

Previous, we supported ingore duplicated keys for UnsafeHashedRelation. We can also optimize LongHashedRelation.
Bug: Row count based shuffle read to optimize global limit after a single partition shuffle (optionally with input partition sorted)
Status: Open | Priority: Major | Resolution: 
Created: 14/Dec/22 02:34
Affects Version/s: 3.4.0

h3. Problem Statement

In current Spark optimizer, a single partition shuffle might be created for a limit if this limit is not the last non-action operation (e.g. a filter following the limit and the data size exceeds a limit). There is a possibility that the previous output partitions before go into this limit are sorted. The single partition shuffle approach has a correctness bug in this case: shuffle read partitions could be out of partition order and the limit exec just take the first limit rows which could lose the order thus result into wrong result. This is a shuffle so it is relatively costly. Meanwhile, to correct this bug, a native solution is to sort all the data fed into limit again, which is another overhead. 

h3. Proposed idea
So we propose a row count based AQE algorithm that optimizes this problem by two folds:

1. Avoid the extra sort on the shuffle read side (or with the limit exec) to achieve the correct result.
2. Avoid reading all shuffle data from mappers for this single partition shuffle to reduce shuffle cost.

Note that 1. is only applied for the sorted partition case where 2. is applied for general single partition shuffle + limit case

 

The algorithm works as the following: 

1. Each mapper will record a row count when writing shuffle data.

2. Since this is single shuffle partition case, there is only one partition but N mappers.

3. A accumulatorV2 is implemented to collect a list of tuple which records the mapping between mapper id and the number of row written by the mapper (row count metrics)

4. AQE framework detects a plan shape of shuffle plus a global limit.

5. AQE framework reads necessary data from mappers based on the limit. For example, if mapper 1 writes 200 rows and mapper 2 writes 300 rows, and the limit is 500, AQE creates shuffle read node to write from both mapper 1 and 2, thus skip the left mappers.

6. This is both correct for limit with the sorted or non-sorted partitions.


Steps to solve this problem:
1. Add a per mapper row count collector
2. Add a AQE rule to use the row count metrics to decide dynamic how many shuffle data from mappers to read.
Bug: Spark doesn't support multiple documents in one pod template file
Status: Open | Priority: Major | Resolution: 
Created: 12/Dec/22 09:17
Affects Version/s: 3.2.2

When defining multiple documents in the pod template file:
{code:java}
apiVersion: v1
kind: Pod
...# the first document to create a PVC
...
--- # use tripple-dash to split multiple documents in one yaml file
apiVersion: v1
kind: Pod
...
... # the second document to create the Spark pod.
{code}
 

Then set the podTemplate file in spark-submit command:

 
{code:java}
--conf spark.kubernetes.executor.podTemplateFile=s3a://***/executor_pod_template.yaml \{code}
 

Then Spark throws:
{code:java}
22/12/09 17:17:41 ERROR KubernetesUtils: Encountered exception while attempting to load initial pod spec from file
expected a single document in the stream
in 'reader', line 1, column 1:
apiVersion: v1
^
but found another document
in 'reader', line 27, column 1:
---
^
at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:125)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:150)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:490)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:429)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshalYaml(Serialization.java:304)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:245)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:201)
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:189)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:656)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:315)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:330)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:86)
at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:105)
{code}
 

The entrypoint code: [https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105]
Bug: MetricsSystem report may cost too much time, which will lead to spark application failed on yarn.
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Dec/22 03:25
Affects Version/s: 2.4.8

My issue is similar to: SPARK-31625（ [https://github.com/apache/spark/pull/28435).]

In the scenario where the shutdown hook does not run (e.g., timeouts, etc.), the application is not unregistered, resulting in YARN RM resubmitting the application even if it succeeded.
{code:java}
22/12/08 09:28:06 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
22/12/08 09:28:06 INFO SparkContext ：Invoking stop() from shut down hook 
22/12/08 09:28:06 INFO SparkContext ：SparkUI ： Stopped Spark web UI at xxx
22/12/08 09:28:16 WARN ShutdownHookManager: ShutdownHook '$anon$2' timeout, java.util.concurrent.TimeoutException java.util.concurrent.TimeoutException     at java.util.concurrent.FutureTask.get(FutureTask.java:205)     at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:67) 22/12/08 09:28:26 WARN ShutdownHookManager: ShutdownHook 'ClientFinalizer' timeout, java.util.concurrent.TimeoutException java.util.concurrent.TimeoutException     at java.util.concurrent.FutureTask.get(FutureTask.java:205)     at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:67) 22/12/08 09:28:36 ERROR ShutdownHookManager: ShutdownHookManger shutdown forcefully. {code}
From the log, it seems that the shutdown hook of SparkContext is hang after the UI is closed. Finally, the hadoop shutdown manager threw a timeout exception and shutdown forcefully.

This eventually led to the Spark Application being marked as FAILED by Yarn, because the unregister in the ApplicationMaster was not executed.

Review the code in SparkContext#stop(), after closed the web UI, it is metricsSystem#report(). However, this method may be blocked for a long time for various reasons (such as network timeout), which is the root cause of the final shutdown hook timeout.

In our scenario, the network is unstable during an abnormal period of time, which causes sinks to take a long time to throw a connection time out exception, which directly causes the SparkContext to fail to stop within 10s.
{code:java}
Utils.tryLogNonFatalError {
  _ui.foreach(_.stop())
}
if (env != null) {
  Utils.tryLogNonFatalError {
    env.metricsSystem.report()
  }
} {code}
 

We use Spark 2. x, and not sure whether Spark 3. x has similar problems. Please someone review it.
Bug: Clarify that input_file_name returns an URI encoded file name
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Dec/22 16:54
Affects Version/s: 3.3.0

The input_file_name function "Returns the name of the file being read, or empty string if not available." ([docs|[https://spark.apache.org/docs/latest/api/sql/#input_file_name]]). However, the returned file name is URI encoded. We should update the documentation to reflect this.
Bug: Spark Thrift Server operation log output is empty
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Dec/22 06:42
Affects Version/s: 3.3.0

when i use jdbc HiveStatement.getQueryLog() to get query log, it return empty.

then i check log on thrift server ,it is also empty.

This bug was introduced in this commit  SPARK-40742  commit id 8e31554bf07cd0f89a45fc393ccc3a77e8a2120d.
Bug: Introduce IsolatedThreadSafeRpcEndpoint to extend IsolatedRpcEndpoint 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/22 06:54
Affects Version/s: 2.4.8

The current RpcEndpoint extension to IsolatedRpcEndpoint always implies that it's thread-safe. However,  it's not explicit to developers. So people may mistakenly override the IsolatedRpcEndpoint#threadCount with more than 1 threadCount leading to the thread-safety broken.
Bug: Improve the performance of try_cast
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/22 05:14
Affects Version/s: 3.4.0

For try_cast and the legacy cast (cast with ANSI SQL mode off), there are conversion which has the same behaviors:
 * cast(string as date/timestamp/TimestampNTZ)
 * cast(string as double/float/decimal)

So, the try_cast can be faster if it uses the evaluation code path of the legacy cast, instead of turning into ANSI cast and catching exceptions.
Bug: PySpark built from master code raise error "java.lang.ClassNotFoundException: org.eclipse.jetty.server.Handler"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/22 13:33
Affects Version/s: 3.4.0

Reproducing steps:

first, clone spark master code, then
 # Build pyspark package by: `./dev/make-distribution.sh --pip` 
 # install built pyspark package
 # run python code: `from pyspark.sql import SparkSession; SparkSession.builder.getOrCreate()`

Got error: "java.lang.ClassNotFoundException: org.eclipse.jetty.server.Handler"

 

The Jetty server should be only used by spark thrift server component, and even if I built with `-Phive-thriftserver` option, the error still happen.