av9ash/GitBugs huggingface dataset:
Bug: SPIP: Row-level operations in Data Source V2
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jun/21 20:21
Affects Version/s: 3.3.0

Row-level operations such as UPDATE, DELETE, MERGE are becoming more and more important for modern Big Data workflows. Use cases include but are not limited to deleting a set of records for regulatory compliance, updating a set of records to fix an issue in the ingestion pipeline, applying changes in a transaction log to a fact table. Row-level operations allow users to easily express their use cases that would otherwise require much more SQL. Common patterns for updating partitions are to read, union, and overwrite or read, diff, and append. Using commands like MERGE, these operations are easier to express and can be more efficient to run.

Hive supports [MERGE|https://blog.cloudera.com/update-hive-tables-easy-way/] and Spark should implement similar support.

SPIP: https://docs.google.com/document/d/12Ywmc47j3l2WF4anG5vL4qlrhT2OKigb7_EbIKhxg60
Bug: Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions 
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/Jun/21 20:40
Affects Version/s: 3.2.0

Currently Spark uses Hadoop shaded client by default. However, if Spark users want to build Spark with older version of Hadoop, such as 3.1.x, the shaded client cannot be used (currently it only support Hadoop 3.2.2+ and 3.3.1+). Therefore, this proposes to offer a new Maven profile "no-shaded-client" for this use case.
Bug: Support Timestamp without time zone data type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/21 07:12
Affects Version/s: 3.4.0

Spark SQL today supports the TIMESTAMP data type. However the semantics provided actually match TIMESTAMP WITH LOCAL TIMEZONE as defined by Oracle. Timestamps embedded in a SQL query or passed through JDBC are presumed to be in session local timezone and cast to UTC before being processed.
 These are desirable semantics in many cases, such as when dealing with calendars.
 In many (more) other cases, such as when dealing with log files it is desirable that the provided timestamps not be altered.
 SQL users expect that they can model either behavior and do so by using TIMESTAMP WITHOUT TIME ZONE for time zone insensitive data and TIMESTAMP WITH LOCAL TIME ZONE for time zone sensitive data.
 Most traditional RDBMS map TIMESTAMP to TIMESTAMP WITHOUT TIME ZONE and will be surprised to see TIMESTAMP WITH LOCAL TIME ZONE, a feature that does not exist in the standard.

In this new feature, we will introduce TIMESTAMP WITH LOCAL TIMEZONE to describe the existing timestamp type and add TIMESTAMP WITHOUT TIME ZONE for standard semantic.
 Using these two types will provide clarity.
 We will also allow users to set the default behavior for TIMESTAMP to either use TIMESTAMP WITH LOCAL TIME ZONE or TIMESTAMP WITHOUT TIME ZONE.
h3. Milestone 1 – Spark Timestamp equivalency ( The new Timestamp type TimestampWithoutTZ meets or exceeds all function of the existing SQL Timestamp):
 * Add a new DataType implementation for TimestampWithoutTZ.
 * Support TimestampWithoutTZ in Dataset/UDF.
 * TimestampWithoutTZ literals
 * TimestampWithoutTZ arithmetic(e.g. TimestampWithoutTZ - TimestampWithoutTZ, TimestampWithoutTZ - Date)
 * Datetime functions/operators: dayofweek, weekofyear, year, etc
 * Cast to and from TimestampWithoutTZ, cast String/Timestamp to TimestampWithoutTZ, cast TimestampWithoutTZ to string (pretty printing)/Timestamp, with the SQL syntax to specify the types
 * Support sorting TimestampWithoutTZ.

h3. Milestone 2 – Persistence:
 * Ability to create tables of type TimestampWithoutTZ
 * Ability to write to common file formats such as Parquet and JSON.
 * INSERT, SELECT, UPDATE, MERGE
 * Discovery

h3. Milestone 3 – Client support
 * JDBC support
 * Hive Thrift server

h3. Milestone 4 – PySpark and Spark R integration
 * Python UDF can take and return TimestampWithoutTZ
 * DataFrame support
Bug: Support repartition expand partitions in AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 09:04
Affects Version/s: 3.2.0

Currently, we don't support expand partition dynamically in AQE which is not friendly for some data skew job.

Let's say if we have a simple query:

{code:java}
SELECT * FROM table DISTRIBUTE BY col
{code}

The column of `col` is skewed, then some shuffle partitions would handle too much data than others.

If we haven't inroduced extra shuffle, we can optimize this case by expanding partitions in AQE.
Bug: Deprecate ps.broadcast API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 04:14
Affects Version/s: 3.2.0

We have [ps.broadcast|https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.broadcast.html] in pandas API on Spark, but it's duplicated with [DataFrame.spark.hint|https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.spark.hint.html] when using this API with "broadcast".

So, we'd better deprecate this and [broadcast|http://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.broadcast.html] function in PySpark as well.
Bug: Mapping the `mode` argument to pandas in DataFrame.to_csv
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 04:03
Affects Version/s: 3.2.0

pandas and pandas-on-Spark both have an argument named `mode` in the [DataFrame.to_csv, |https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_csv.html]but the acceptable strings are different.

pandas can accept the "w", "w+", "a", "a+" where as pandas-on-Spark can accept "append", "overwrite", "ignore", "error" or "errorifexists".

We should map these acceptable strings to pandas.

e.g. "w" will work as Spark's "overwrite". In addition, mode can take Spark's "overwrite" too.
Bug: Spark read write with multiple Hadoop HA cluster limitation
Status: Open | Priority: Major | Resolution: 
Created: 18/Jun/21 16:45
Affects Version/s: 2.4.3

I have two Hadoop HA cluster: h1 and h2. Want to read from h1-HDFS and write to h2-HDFS using spark. As both HDFS are in HA, so need to set spark hadoop configuration with HDFS details
{code:java}
spark.sparkContext().hadoopConfiguration().set(<HADOOP_RPC_ADDRESS_AND_DETAILS>){code}
So with a single spark session job one of the Hadoop configuration will overwrite with write details and will try to read from that configuration, resulting in no file/path found.

Similar thing will happen with HDFS to external Hive write(I am writing in external Hive table owned HDFS), but more keen on above problem solution
Bug: Does anyone has performance issue after upgrade from 3.0 to 3.1?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 16/Jun/21 13:00
Affects Version/s: 3.1.2

Hello.

 

We had using spark 3.0.2 and query was executed in ~100 seconds.

After we upgraded Spark to 3.1.1 (tried also 3.1.2 - same, slow performance) - our query execution time started taking ~260 seconds it is huge increase 250-300 % of execution time increase.

 

We tried quite simple query.

In query we using UDF (*org.apache.spark.sql.functions*)

) - which explodes data and do polygon hit test. Nothing changed in our code from query perspective.
 It is 1 VM box cluster

 

Maybe anyone faced similar issue?

Attached some details from spark dashboard.

 

*Looks like it is UDF related slowdown. As queries which does not use UDF`s performance is same and which uses UDFs - starting from 3.1 performance decreased.*
Bug: Spark SQL does not support creating views using DataSource v2 based data sources
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 01:24
Affects Version/s: 2.4.8

When a temporary view is created in Spark SQL using an external data source, Spark then tries to create the relevant relation using DataSource.resolveRelation() method. Unlike DataFrameReader.load(), resolveRelation() does not check if the provided DataSource implements the DataSourceV2 interface and instead tries to use the RelationProvider trait in order to generate the Relation.

Furthermore, DataSourceV2Relation is not a subclass of BaseRelation, so it cannot be used in resolveRelation().

Last, I tried to implement the RelationProvider trait in my Java implementation of DataSourceV2, but the match inside resolveRelation() did not detect it as RelationProvider.
Bug: can't read external hive table on spark
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 18/Jun/21 02:49
Affects Version/s: 2.3.2

I create a external hive table with hdfs file which is formatted as json string.

I can read the data field of this hive table with the help of  org.apache.hive.hcatalog.data.JsonSerDe which is packed in hive-hcatalog-core.jar in hive shell.

But when I try to use the spark (pyspark ,spark-shell or whatever) ,I just can't read it.

It gave me a error  Table: Unable to get field from serde: org.apache.hive.hcatalog.data.JsonSerDe

I've copied the jar (hive-hcatalog-core.jar) to $spark_home/jars and yarn libs and rerun ,there is no effect,even use --jars $jar_path/hive-hcatalog-core.jar.But  when I browse the webpage of spark task ,I can actually find the jar in the env list.
Bug: Deprecate DataFrame.to_spark_io
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 04:16
Affects Version/s: 3.2.0

We should deprecate the [DataFrame.to_spark_io|https://docs.google.com/document/d/1RxvQJVf736Vg9XU7uiCaRlNl-P7GdmFGa6U3Ab78JJk/edit#heading=h.todz8y4xdqrx] since it's duplicated with [DataFrame.spark.to_spark_io|https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.spark.to_spark_io.html], and it's not existed in pandas.
Bug: Use the same cleanup logic as Py4J in inheritable thread API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/21 01:07
Affects Version/s: 3.2.0

After https://github.com/apache/spark/commit/6d309914df422d9f0c96edfd37924ecb8f29e3a9, the test became flaky:

{code}
======================================================================
ERROR [71.813s]: test_save_load_pipeline_estimator (pyspark.ml.tests.test_tuning.CrossValidatorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/ml/tests/test_tuning.py", line 589, in test_save_load_pipeline_estimator
    self._run_test_save_load_pipeline_estimator(DummyLogisticRegression)
  File "/__w/spark/spark/python/pyspark/ml/tests/test_tuning.py", line 572, in _run_test_save_load_pipeline_estimator
    cvModel2 = crossval2.fit(training)
  File "/__w/spark/spark/python/pyspark/ml/base.py", line 161, in fit
    return self._fit(dataset)
  File "/__w/spark/spark/python/pyspark/ml/tuning.py", line 747, in _fit
    bestModel = est.fit(dataset, epm[bestIndex])
  File "/__w/spark/spark/python/pyspark/ml/base.py", line 159, in fit
    return self.copy(params)._fit(dataset)
  File "/__w/spark/spark/python/pyspark/ml/pipeline.py", line 114, in _fit
    model = stage.fit(dataset)
  File "/__w/spark/spark/python/pyspark/ml/base.py", line 161, in fit
    return self._fit(dataset)
  File "/__w/spark/spark/python/pyspark/ml/pipeline.py", line 114, in _fit
    model = stage.fit(dataset)
  File "/__w/spark/spark/python/pyspark/ml/base.py", line 161, in fit
    return self._fit(dataset)
  File "/__w/spark/spark/python/pyspark/ml/classification.py", line 2924, in _fit
    models = pool.map(inheritable_thread_target(trainSingleClass), range(numClasses))
  File "/__t/Python/3.6.13/x64/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/__t/Python/3.6.13/x64/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/__t/Python/3.6.13/x64/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/__t/Python/3.6.13/x64/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/__w/spark/spark/python/pyspark/util.py", line 324, in wrapped
    InheritableThread._clean_py4j_conn_for_current_thread()
  File "/__w/spark/spark/python/pyspark/util.py", line 389, in _clean_py4j_conn_for_current_thread
    del connections[i]
IndexError: deque index out of range

----------------------------------------------------------------------
{code}
Bug: Fix Index.difference to avoid collect 'other' to driver side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 01:06
Affects Version/s: 3.2.0

See:
https://github.com/databricks/koalas/pull/1325#discussion_r647889901
https://github.com/databricks/koalas/pull/1325#discussion_r647890007

{code}
midx1 = ps.MultiIndex.from_tuples([('a', 'x', 1), ('b', 'z', 2), ('k', 'z', 3)])
midx1.difference(idx1)
{code}

{code}
pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Index.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.
{code}

In addition, calling {{MultiIndex.from_tuples}} will result in collecting all into driver side.
Bug: Use type-annotation based pandas_udf or avoid specifying udf types to suppress warnings.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/21 22:49
Affects Version/s: 3.2.0

The usage of {{pandas_udf}} in pandas-on-Spark is outdated and shows warnings.
We should use type-annotation based {{pandas_udf}} or avoid specifying udf types.
Bug: Use higher PyArrow in GitHub Actions build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/21 14:23
Affects Version/s: 3.2.0

We set the upperbound of PyArrow at SPARK-33190 that's fixed in SPARK-33189. With PyArrow 4.x, all PySpark tests pass too. So it should better test higher versions which more people use.
Bug: Rename "pandas APIs on Spark" to "pandas API on Spark" in the documents
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/21 03:17
Affects Version/s: 3.2.0

The project name is fixed to "pandas API on Spark" from "pandas APIs on Spark", we should rename the "pandas APIs on Spark" to "pandas API on Spark"
Bug: Serie to Scalar pandas_udf in GroupedData.agg() breaks the following monotonically_increasing_id()
Status: Reopened | Priority: Major | Resolution: 
Created: 11/Jun/21 21:21
Affects Version/s: 3.1.1

Hello,

I encountered an issue when using a Serie to Scalar `{{panda_udf}}` in `{{GroupedData.agg()}}` followed by `{{monotonically_increasing_id()}}`. I obtain duplicated ids. Actually, the partition offset in the id seems to be zero on all partitions. The problem is avoided by using `{{asNondeterministic}}`.

Minimal reproducing example
{code:java}
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from pyspark.sql.functions import pandas_udf
import pandas as pd
from pyspark.sql.types import IntegerType

spark = SparkSession.builder\
.config("spark.sql.execution.arrow.pyspark.enabled", "true")\
.config("spark.sql.shuffle.partitions", "8")\
.master("local[4]").getOrCreate()

@pandas_udf(IntegerType())
def sum_pandas(vals: pd.Series) -> int:
    return int(vals.to_numpy().sum())

@pandas_udf(IntegerType())
def sum_pandas2(vals: pd.Series) -> int: 
    return int(vals.to_numpy().sum())

sum_pandas2 = sum_pandas2.asNondeterministic()

l = [(i%100,i) for i in range(2000)]

data = spark.createDataFrame(l, schema=["col1","col2"])
data.groupby("col1").agg(sum_pandas("col2").alias("sum"))\
.withColumn("group_id", F.monotonically_increasing_id()).show()

data = spark.createDataFrame(l, schema=["col1","col2"])
data.groupby("col1").agg(sum_pandas2("col2").alias("sum"))\
.withColumn("group_id", F.monotonically_increasing_id()).show(){code}
Output
{code:java}
+----+-----+--------+
|col1|  sum|group_id|
+----+-----+--------+
|   2|19040|       0|
|  12|19240|       1|
|  26|19520|       2|
|  28|19560|       3|
|  29|19580|       4|
|  30|19600|       5|
|  33|19660|       6|
|  42|19840|       7|
|  48|19960|       8|
|  67|20340|       9|
|  73|20460|      10|
|  88|20760|      11|
|  91|20820|      12|
|  93|20860|      13|
|   9|19180|       0|
|  11|19220|       1|
|  22|19440|       2|
|  32|19640|       3|
|  36|19720|       4|
|  40|19800|       5|
+----+-----+--------+
only showing top 20 rows

+----+-----+----------+
|col1|  sum|  group_id|
+----+-----+----------+
|   2|19040|         0|
|  12|19240|         1|
|  26|19520|         2|
|  28|19560|         3|
|  29|19580|         4|
|  30|19600|         5|
|  33|19660|         6|
|  42|19840|         7|
|  48|19960|         8|
|  67|20340|         9|
|  73|20460|        10|
|  88|20760|        11|
|  91|20820|        12|
|  93|20860|        13|
|   9|19180|8589934592|
|  11|19220|8589934593|
|  22|19440|8589934594|
|  32|19640|8589934595|
|  36|19720|8589934596|
|  40|19800|8589934597|
+----+-----+----------+
only showing top 20 rows
{code}
Bug: Adjust pandas-on-spark `test_groupby_multiindex_columns` test for different pandas versions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 22:50
Affects Version/s: 3.2.0

Adjust pandas-on-spark test_groupby_multiindex_columns test in order to pass with different pandas versions.

pandas had introduced bugs as below:

- For pandas 1.1.3 and 1.1.4

Type error: only integer scalar arrays can be converted to a scalar index

- For pandas < 1.0.4

Type error: Can only tuple-index with a MultiIndex
Bug: Schema inference of nested structs defaults to map
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/21 07:49
Affects Version/s: 3.2.0

Inferring schema for struct columns causing schema issues as below.
{code:java}
data = [{"inside_struct": {"payment": 100.5, "name": "Lee"}}]
df = spark.createDataFrame(data)
df.show()
+--------------------+
|       inside_struct|
+--------------------+
|{name -> null, pa...|
+--------------------+
{code}
The "inside_struct" is a map, and the "name" column inside of it becomes null.

The schema inferring might decide on a map type with a value type of the first field of the struct, we should fix it.
Bug: Bump up mypy version in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 01:08
Affects Version/s: 3.2.0

The mypy version was pinned at https://issues.apache.org/jira/browse/SPARK-35682 but not the latest version because of the errors below:

{code}
python/pyspark/pandas/indexes/base.py:2007: error: Argument 1 to "from_tuples" of "MultiIndex" has incompatible type "Index"; expected "List[Tuple[Any, ...]]"
python/pyspark/testing/pandasutils.py:41: error: Library stubs not installed for "tabulate" (or incompatible with Python 3.6)
python/pyspark/testing/pandasutils.py:41: note: Hint: "python3 -m pip install types-tabulate"
python/pyspark/testing/pandasutils.py:41: note: (or run "mypy --install-types" to install all missing stub packages)
python/pyspark/testing/pandasutils.py:41: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 2 errors in 2 files (checked 312 source files)
{code}

This is blocked by https://issues.apache.org/jira/browse/SPARK-35683
Bug: Use "concurrency" setting on Github Actions
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Jun/21 13:23
Affects Version/s: 3.1.2

We are using [cancel_duplicate_workflow_runs|https://github.com/apache/spark/blob/a70e66ecfa638cacc99b4e9a7c464e41ec92ad30/.github/workflows/cancel_duplicate_workflow_runs.yml#L1] job to cancel previous jobs when a new job is queued. Now, it has been supported by the github action by using ["concurrency"|https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#concurrency] syntax to make sure only a single job or workflow using the same concurrency group.

related: https://github.com/apache/arrow/pull/10416 and https://github.com/potiuk/cancel-workflow-runs
Bug: Introduce InternalField to manage dtypes and StructFields.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/21 22:11
Affects Version/s: 3.2.0

Currently there are some performance issues in the pandas-on-Spark layer.

One of them is accessing Java DataFrame and run analysis phase too many times, especially just for retrieving the current column names or data types.

We should reduce the amount of unnecessary access.
Bug: createDataFrame fails while to_spark works.
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 06/Jun/21 07:19
Affects Version/s: 3.2.0

Sample code:

{{kdf = ks.DataFrame({}}
{{ 'a': [1, 2, 3],}}
{{ 'b': [2., 3., 4.],}}
{{ 'c': ['string1', 'string2', 'string3'],}}
{{ 'd': [date(2000, 1, 1), date(2000, 2, 1), date(2000, 3, 1)],}}
{{ 'e': [datetime(2000, 1, 1, 12, 0), datetime(2000, 1, 2, 12, 0), datetime(2000, 1, 3, 12, 0)]}}
{{ })}}{{df = kdf.to_spark() # WORKS}}
{{ df = spark.createDataFrame(kdf) # FAILED}}

Error:

{{TypeError: Can not infer schema for type: <class 'str'>}}
Bug: Introduce a way to compare series of array for older pandas
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/21 16:42
Affects Version/s: 3.2.0

PySpark tests ComplexOpsTest.test_add failed with older pandas e.g. v1.0.1, with the ValueError The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().

We need to introduce a way to check the equality when the data are arrays.
Bug: Do not push down extract value in higher order function that references both sides of a join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/21 18:04
Affects Version/s: 3.1.0

Currently, lambda keys can be referenced outside of the lambda function:

{quote}Project [transform(keys#0, lambdafunction(_extract_v1#0, lambda key#0, false)) AS a#0]
+- 'Join Cross
    :- Project [kvs#0[lambda key#0].v1 AS _extract_v1#0]
    :  +- LocalRelation <empty>, [kvs#0]
    +- LocalRelation <empty>, [keys#0]{quote}

This should be unchanged from the original state:

{quote}Project [transform(keys#418, lambdafunction(kvs#417[lambda key#420].v1, lambda key#420, false)) AS a#419]
+- Join Cross
    :- LocalRelation <empty>, [kvs#417]
    +- LocalRelation <empty>, [keys#418]{quote}
Bug: Cleanup the version logic from the pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/21 07:24
Affects Version/s: 3.2.0

There are version checking logic in the pyspark.pandas' __init__.py script, but we don't need to have this anymore since it's ported into PySpark, and we should follow the PySpark's version check logic.
Bug: Remove # noqa in Python API documents.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/21 01:40
Affects Version/s: 3.2.0

There are some unnecessary words "# noqa" are exposed in the Python API documentation.

 

For example, [https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameReader.parquet.html#pyspark.sql.DataFrameReader.parquet.]

 

We should remove this.
Bug: Path level discover for python unittests
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Jun/21 05:07
Affects Version/s: 3.2.0

Now we need to specify the python test cases by manually when we add a new testcase. Sometime, we forgot to add the testcase to module list, the testcase would not be executed.

Such as:
 * pyspark-core pyspark.tests.test_pin_thread

Thus we need some auto-discover way to find all testcase rather than specified every case by manually.