av9ash/GitBugs huggingface dataset:
Bug: Porting "pandas API on Spark: Internals" to PySpark docs.
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 19/Oct/21 05:20
Affects Version/s: 3.2.0

We have a [documents|https://docs.google.com/document/d/1PR88p6yMHIeSxkDkSqCxLofkcnP0YtwQ2tETfyAWLQQ/edit?usp=sharing] for pandas API on Spark internal features, apart from the PySpark official documents.

 

Since pandas API on Spark is officially released in Spark 3.2, it's good to port this internal documents to the PySpark official documents.
Bug: Test type hints against examples
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/21 10:44
Affects Version/s: 3.3.0

Next to tests, examples are the largest piece of code showing actual usage of public API, that we have. 

Originally, {{pyspark-stubs}} tested {{sql}} and {{ml}} examples, but these tests weren't ported during the initial migration.

We should add these back, and extend to {{mllib}}, {{streaming}} and core files.
Bug: Unify v1 and v2 ALTER TABLE .. ADD COLUMNS tests
Status: Open | Priority: Major | Resolution: 
Created: 18/Oct/21 16:59
Affects Version/s: 3.3.0

Extract ALTER TABLE .. ADD COLUMNS tests to the common place to run them for V1 and v2 datasources. Some tests can be places to V1 and V2 specific test suites.
Bug: Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/21 06:44
Affects Version/s: 3.2.0

The assertion below in MapOutoutputTracker.getMapSizesByExecutorId is not guaranteed
{code:java}
assert(mapSizesByExecutorId.enableBatchFetch == true){code}
The reason is during some stage retry cases, the shuffleDependency.shuffleMergeEnabled is set to false, but there will be mergeStatus since the Driver has collected the merged status for its shuffle dependency. If this is the case, the current implementation would set the enableBatchFetch to false, since there are mergeStatus.

Details can be found here:

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L1492]

We should improve the implementation here.
Bug: Avoid executor state sync attempt fail continuously in a short timeframe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Oct/21 00:47
Affects Version/s: 3.2.0

A worker would retry sending the ExecutorStateChanged message when the previous attempt failed. This would not be an issue when the attempt failed with TimeoutException. But if the connection between the worker and the master is broken, the attempt would fail immediately, leading to the retry attempt also fail, and quickly reaches the max attempt limitation.
Bug: Annotations for pyspark.sql.context.SQLContext.createDataFrame are broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/21 20:00
Affects Version/s: 3.3.0

During migration from stubs to inline annotations, variants taking {{RDD}} where incorrectly remove. As a result

 
{code:python}
from pyspark.sql import SQLContext, SparkSession
from pyspark import SparkContext

sc = SparkContext.getOrCreate()
sqlContext= SQLContext(sc)

sqlContext.createDataFrame(sc.parallelize([(1, 2)]))
{code}

although valid, no longer type checks


{code}
main.py:7: error: No overload variant of "createDataFrame" of "SQLContext" matches argument type "RDD[Tuple[int, int]]"
main.py:7: note: Possible overload variants:
main.py:7: note:     def [RowLike in (List[Any], Tuple[Any, ...], Row)] createDataFrame(self, data: Iterable[RowLike], samplingRatio: Optional[float] = ...) -> DataFrame
main.py:7: note:     def [RowLike in (List[Any], Tuple[Any, ...], Row)] createDataFrame(self, data: Iterable[RowLike], schema: Union[List[str], Tuple[str, ...]] = ..., verifySchema: bool = ...) -> DataFrame
main.py:7: note:     def createDataFrame(self, data: DataFrameLike, samplingRatio: Optional[float] = ...) -> DataFrame
main.py:7: note:     <3 more non-matching overloads not shown>
Found 1 error in 1 file (checked 1 source file)
{code}
Bug: What's the progress of vectorized execution for spark?
Status: Open | Priority: Major | Resolution: 
Created: 18/Oct/21 04:04
Affects Version/s: 3.2.0

Spark has support vectorized read for ORC and parquet. What's the progress of other vectorized execution, e.g. vectorized write,  join, aggr, simple operator (string function, math function)? 

Hive support vectorized execution in early version (https://cwiki.apache.org/confluence/display/hive/vectorized+query+execution) As we know, Spark is replacement of Hive. I guess the reason why Spark does not support vectorized execution maybe the difficulty of design or implementation in Spark is larger. What's the main issue for Spark to support vectorized execution?
Bug: Push down filters having both partitioning and non-partitioning columns
Status: Open | Priority: Major | Resolution: 
Created: 31/Oct/21 13:46
Affects Version/s: 3.2.0

Currently, filters having both partitioning and non-partitioning columns are lost during the creation of {{FileSourceScanExec}} and not pushed down to the data source. However, theoretically and practically, there is no reason to exclude such filters from {{dataFilters}}. For any partitioned source data file, the values of partitioning columns are the same for all rows. They can be stored physically (or reconstructed logically) along with statistics for non-partitioning columns to allow more powerful data skipping. If a data source doesn't know how to handle such filters, it can simply ignore such filters.

Example: Suppose that there is a table {{MYTAB}} with two columns {{A}} and {{B}}, partitioned by {{A}} (Hive partitioning). Currently, data skipping cannot be applied to queries like {{select * from MYTAB where A < B + 7}} because {{A < B + 7}} is included in neither {{partitionFilters}} nor {{dataFilters}}. However, we could have included the filter in {{dataFilters}} because data sources have no obligation to use {{dataFilters}} and they could have ignored filters that they cannot use.

It's not obvious whether we can change the semantics of {{FileSourceScanExec.dataFilters}} without breaking existing code. It is passed to {{FileIndex.listFiles}} and {{FileFormat.buildReaderWithPartitionValues}} and the contracts for the methods are not clear enough.

If we should not change {{dataFilters}}, we might have to add a new member variable to {{FileSourceScanExec}} (e.g. {{dataFiltersWithPartitionColumns}}) and add an overload of {{listFiles}} to the {{FileIndex}} trait, which defaults to the existing {{listFiles}} without using the filters. Both {{dataFilters}} and {{dataFiltersWIthoutPartitionColumns}} are optional; implementations can ignore the filters if they can't utilize them.
Bug: Support RowToColumnarExec with Arrow
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Oct/21 00:56
Affects Version/s: 3.2.0

This Jira is aim to support Arrow format in RowToColumnarExec 

Current ArrowColumnVector is not fully equivalent to OnHeap/OffHeapColumnVector in spark, so RowToColumnarExec doesn't support write to Arrow format so far.

since Arrow API is now being more stable, and using pandas udf will perform much better than python udf.

I am  proposing to support RowToColumnarExec with Arrow.

What I did in this PR is to add a load api in ArrowColumnVector to load arrowRecordBatch to ArrowColumnVector, then called inside RowToColumnarExec doExecute.

 

UTs are also added to test this new API and RowToColumnarExec with ArrowFormat
Bug: RowToColumnConverter  support AnsiIntervalType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Oct/21 10:00
Affects Version/s: 3.3.0

currently, we have RowToColumnConverter for all data types except AnsiIntervalType
{code:java}
// code placeholder
val core = dataType match {
  case BinaryType => BinaryConverter
  case BooleanType => BooleanConverter
  case ByteType => ByteConverter
  case ShortType => ShortConverter
  case IntegerType | DateType => IntConverter
  case FloatType => FloatConverter
  case LongType | TimestampType => LongConverter
  case DoubleType => DoubleConverter
  case StringType => StringConverter
  case CalendarIntervalType => CalendarConverter
  case at: ArrayType => ArrayConverter(getConverterForType(at.elementType, at.containsNull))
  case st: StructType => new StructConverter(st.fields.map(
    (f) => getConverterForType(f.dataType, f.nullable)))
  case dt: DecimalType => new DecimalConverter(dt)
  case mt: MapType => MapConverter(getConverterForType(mt.keyType, nullable = false),
    getConverterForType(mt.valueType, mt.valueContainsNull))
  case unknown => throw QueryExecutionErrors.unsupportedDataTypeError(unknown.toString)
}

if (nullable) {
  dataType match {
    case CalendarIntervalType => new StructNullableTypeConverter(core)
    case st: StructType => new StructNullableTypeConverter(core)
    case _ => new BasicNullableTypeConverter(core)
  }
} else {
  core
}

{code}
Bug: MetricsReporter producing NullPointerException when element 'triggerExecution' not present in Map[]
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/21 16:27
Affects Version/s: 3.2.0

The exception occurs in MetricsReporter when it tries to register gauges using lastProgress of each stream. This problem was partially fixed in https://issues.apache.org/jira/browse/SPARK-22975 but in introduced a NPE when 'triggerExecution' element is not present in durationMS map:

 
{code:java}
registerGauge("latency", _.durationMs.get("triggerExecution").longValue(), 0L)
{code}
 

I find it difficult to reproduce this every time but it happens every few restarts when structured streaming uses a slow event source (very little or no events). In my case it breaks metric reporting via Codehale/Dropwizard and generates multiple stacktraces such as:
{code:java}
21/09/16 09:51:36 ERROR ScheduledReporter: Exception thrown from GangliaReporter#report. Exception was suppressed.
java.lang.NullPointerException
	at org.apache.spark.sql.execution.streaming.MetricsReporter.$anonfun$new$3(MetricsReporter.scala:43)
	at org.apache.spark.sql.execution.streaming.MetricsReporter.$anonfun$new$3$adapted(MetricsReporter.scala:43)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.sql.execution.streaming.MetricsReporter$$anon$1.getValue(MetricsReporter.scala:68)
	at com.codahale.metrics.ganglia.GangliaReporter.reportGauge(GangliaReporter.java:353)
	at com.codahale.metrics.ganglia.GangliaReporter.report(GangliaReporter.java:240)
	at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:237)
	at com.codahale.metrics.ScheduledReporter.lambda$start$0(ScheduledReporter.java:177)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{code}
I'm happy to implement a fix.
Bug: Disallow casting Date as Numeric types
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 29/Oct/21 12:01
Affects Version/s: 3.3.0

Currently, Date type values can be cast as Numeric types. However, the result is always NULL.
On the other hand, Numeric values can't be cast as Date type.
It doesn't make sense to keep the behavior of casting Date to null numeric types. I suggest to disallow the conversion. We can have a legacy flag `spark.sql.legacy.allowCastDateAsNumeric` if users really want to fall back to the legacy behavior.
Bug: Can't read files in one of Parquet encryption modes (external keymaterial) 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/21 09:05
Affects Version/s: 3.2.0

Parquet encryption has a number of modes. One of them is "external keymaterial", which keeps encrypted data keys in a separate file (as opposed to inside Parquet file). Upon reading, the Spark Parquet connector does not pass the file path, which causes an NPE.
Bug: Pass all UTs in `sql/hive` with Java 17
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/21 03:55
Affects Version/s: 3.3.0

Add `extraJavaTestArgs` to sql/hive pom.xml and run 
{code:java}
build/mvn clean install -Phadoop-3.2 -Phive-2.3 -Phive -pl sql/hive
{code}
there are 22 failed tests 
{code:java}
Run completed in 1 hour, 2 minutes, 3 seconds.
Total number of tests run: 3547
Suites: completed 117, aborted 0
Tests: succeeded 3525, failed 22, canceled 6, ignored 605, pending 0
*** 22 TESTS FAILED ***
{code}
Bug: parse_url can not handle `{` and `}` correctly
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 26/Oct/21 11:56
Affects Version/s: 2.4.8

when we execute the follow sql command
{code:java}
select parse_url('http://facebook.com/path/p1.php?query={aa}', 'QUERY')
{code}
the expected result:

    query=\{aa}

the actual result:

    null
Bug: Support AnsiInterval radix sort
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/21 03:59
Affects Version/s: 3.3.0

The radix sort is more faster than timsort, the benchmark result can see in `SortBenchmark`.

Since the `AnsiInterval` data type is comparable:
- `YearMonthIntervalType` -> int ordering
- `DayTimeIntervalType` -> long ordering

And we aslo support radix sort when the ordering column date type is int or long.

So `AnsiInterval` radix sort can be supported.
Bug: Support non-literal frame bound value for window functions
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/21 04:08
Affects Version/s: 3.0.3


{code:sql}
sql("""
with va as (
select 15 a, 100 b
 union all
select 15, 120
 union all
select 15, 130
 union all
select 15, 150
)
select t.*, 
       min(t.b) over(partition by t.a order by t.b range between 0.15*t.b preceding and current row) c 
  from va t 
""").show
{code}

throws 


{code:java}
org.apache.spark.sql.catalyst.parser.ParseException:
Frame bound value must be a literal.(line 12, pos 65)
{code}

The non-literal expression *0.15*t.b* might leads this exception.

But the non-literal frame bound value has already been support by oracle which is very useful.
Bug: Support Writable ArrowColumnarVector
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/21 00:55
Affects Version/s: 3.2.0

This Jira is aim to add Arrow format as an alternative for ColumnVector solution.

Current ArrowColumnVector is not fully equivalent to OnHeap/OffHeapColumnVector in spark, and since Arrow API is now being more stable, and using pandas udf will perform much better than python udf.

I am  proposing to fully support arrow format as an alternative to ColumnVector just like the other two.

What I did in this PR is to create a new class in the same package with OnHeap/OffHeapColumnVector and extend from WritableColumnVector to support all put APIs.

UTs are covering all Data Format with testing on writing to columnVector and reading from columnVector. I also added 3 UTs for testing on loading from ArrowRecordBatch and allocateColumns .
Bug: Alter table properties should invalidate cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/21 09:08
Affects Version/s: 3.0.3

The table properties can change the behavior of wriing. e.g. the parquet table with `parquet.compression`.

If you execute the following SQL, we will get the file with snappy compression rather than zstd.
{code:java}
CREATE TABLE t (c int) STORED AS PARQUET;
// cache table metadata
SELECT * FROM t;
ALTER TABLE t SET TBLPROPERTIES('parquet.compression'='zstd');
INSERT INTO TABLE t values(1);
{code}
So we should invalidate the table cache after alter table properties.
Bug: executorIdleTimeout is not working for pending pods on K8s
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/21 23:08
Affects Version/s: 3.1.0

SPARK-33099 added the support to respect "spark.dynamicAllocation.executorIdleTimeout" in ExecutorPodsAllocator. However, when it checks if a pending executor pod is timed out, it checks against the pod's "startTime". A pending pod "startTime" is empty, and this causes the function "isExecutorIdleTimedOut()" always return true for pending pods.

This caused the issue, pending pods are deleted immediately when a stage is finished and several new pods got recreated again in the next stage.
Bug: Cancel all running job after AQE plan finished
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Oct/21 12:02
Affects Version/s: 3.3.0

We see stage was still running after AQE plan finished. This is because the plan which contains a empty join has been converted to `LocalTableScanExec` during `AQEOptimizer`, but the other side of this join is still running (shuffle map stage).

 

It's no meaning to keep running the stage, It's better to cancel the running stage after AQE plan finished in case wasting the task resource.
Bug: Upgrade flake8 to 3.9.0 or above in Jenkins
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/21 00:50
Affects Version/s: 3.3.0

In flake8 < 3.9.0, F401 error occurs for imports when the imported identities are used in a {{bound}} argument in {{TypeVar(..., bound="XXX")}}.

For example:

{code:python}
if TYPE_CHECKING:
    from pyspark.pandas.base import IndexOpsMixin

IndexOpsLike = TypeVar("IndexOpsLike", bound="IndexOpsMixin")
{code}


Since this behavior is fixed In flake8 >= 3.9.0, we should upgrade the flake8 installed in Jenkins to 3.9.0 or above.
And also we might update the {{MINIMUM_FLAKE8}} in the {{lint-python}} from 3.8.0 to 3.9.0.
Bug: Pass all UTs in `yarn` with Java 17
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/21 03:55
Affects Version/s: 3.3.0

{code:java}
build/mvn clean install -DskipTests -pl resource-managers/yarn -am -Pyarn
build/mvn test -pl resource-managers/yarn -Pyarn
{code}
{code:java}
Run completed in 5 minutes, 26 seconds.
Total number of tests run: 144
Suites: completed 19, aborted 0
Tests: succeeded 131, failed 13, canceled 1, ignored 0, pending 0
*** 13 TESTS FAILED ***
{code}
Bug: MapStatus adds localDirs to avoid the rpc request by method getHostLocalDirs when shuffle reading
Status: Open | Priority: Major | Resolution: 
Created: 14/Oct/21 09:08
Affects Version/s: 3.1.2

When executing the ShuffleBlockFetcherIterator.fetchHostLocalBlocks method, in order to obtain the hostLocalDirs value, we need to send an RPC request through ExternalBlockStoreClient or NettyBlockTransferService. Then get shuffle data according to blockId and localDirs.

We can add localDir to the BlockManagerId class of MapStatus, so that we can get localDir directly when fetch host-local blocks without sending RPC requests.

The benefits are:
1. No need to send RPC request localDirs value when fetchHostLocalBlocks;
2. When the external shuffle service is enabled, there is no need to register ExecutorShuffleInfo in the ExternalShuffleBlockResolver class, nor to save the ExecutorShuffleInfo data in the ExternalShuffleBlockResolver class through leveldb.
3. Also, there is no need to cache host-local dirs in the HostLocalDirManager class.
Bug: Publicise UpperCaseCharStream
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Oct/21 04:55
Affects Version/s: 2.2.3

Many Spark extension projects are copying `UpperCaseCharStream` because it is private beneath `parser` package, such as:

[Delta Lake|https://github.com/delta-io/delta/blob/625de3b305f109441ad04b20dba91dd6c4e1d78e/core/src/main/scala/io/delta/sql/parser/DeltaSqlParser.scala#L290]

[Hudi|https://github.com/apache/hudi/blob/3f8ca1a3552bb866163d3b1648f68d9c4824e21d/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/parser/HoodieCommonSqlParser.scala#L112]

[Iceberg|https://github.com/apache/iceberg/blob/c3ac4c6ca74a0013b4705d5bd5d17fade8e6f499/spark3-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala#L175]

[Submarine|https://github.com/apache/submarine/blob/2faebb8efd69833853f62d89b4f1fea1b1718148/submarine-security/spark-security/src/main/scala/org/apache/submarine/spark/security/parser/UpperCaseCharStream.scala#L31]

[Kyuubi|https://github.com/apache/incubator-kyuubi/blob/8a5134e3223844714fc58833a6859d4df5b68d57/dev/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/zorder/ZorderSparkSqlExtensionsParserBase.scala#L108]

[Spark-ACID|https://github.com/qubole/spark-acid/blob/19bd6db757677c40f448e85c74d9995ba97d5942/src/main/scala/com/qubole/spark/datasources/hiveacid/sql/catalyst/parser/ParseDriver.scala#L13]

We can publicise `UpperCaseCharStream` to eliminate code duplication.
Bug: yarn-cluster mode, unregister timeout cause spark retry but AM container exit with code 0
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Oct/21 07:46
Affects Version/s: 3.2.0


1. Cluster mode AM shutdown hook triggered
2. am unregister from RM timeout, but AM shutdown hook have try catch, so AM container exit with code 0.
3. Since RM lose connection with AM, then treat this container as failed.
4. Then client side got application report as final status failed but am container exit code 0. Then retry.
Bug: Add Spark error classes to error message and enforce test coverage
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Oct/21 00:45
Affects Version/s: 3.3.0

We should add Spark error classes to the error message and make sure that all error classes are tested. This will help us understand the error cases, remove dead code, and improve the error messages and classes as we refactor the error messages.
Bug: Python UDF after off-heap vectorized reader can cause crash due to use-after-free in writer thread
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/21 14:57
Affects Version/s: 3.0.3

Python UDFs in Spark SQL are run in a separate Python process. The Python process is fed input by a dedicated thread (`BasePythonRunner.WriterThread`). This writer thread drives the child plan by pulling rows from its output iterator and serializing them across a socket.

When the child exec node is the off-heap vectorized Parquet reader, these rows are backed by off-heap memory. The child node uses a task completion listener to free the off-heap memory at the end of the task, which invalidates the output iterator and any rows it has produced. Since task completion listeners are registered bottom-up and executed in reverse order of registration, this is safe as long as an exec node never accesses its input after its task completion listener has executed.

The BasePythonRunner task completion listener violates this assumption. It interrupts the writer thread, but does not wait for it to exit. This causes a race condition that can lead to an executor crash:
1. The Python writer thread is processing a row backed by off-heap memory.
2. The task finishes, for example because it has reached a row limit.
3. The BasePythonRunner task completion listener sets the interrupt status of the writer thread, but the writer thread does not check it immediately.
4. The child plan's task completion listener frees its off-heap memory, invalidating the row that the Python writer thread is processing.
5. The Python writer thread attempts to access the invalidated row. The use-after-free triggers a segfault that crashes the executor.

https://issues.apache.org/jira/browse/SPARK-33277 describes the same issue, but the fix was incomplete. It did not address the situation where the Python writer thread accesses a freed row.