av9ash/GitBugs huggingface dataset:
Bug: PostgresDialect Doesn't handle arrays of custom data types after postgresql driver version 42.2.22
Status: Open | Priority: Minor | Resolution: 
Created: 09/Aug/22 20:32
Affects Version/s: 3.1.1

Starting in version 42.2.23 (also 42.3.x and 42.4.x), the sql type returned by the postgresql driver is now `ARRAY` with columns with an array of custom data types (e.g. an array of enums). Prior to this version the driver returned the type `CUSTOM`. PostgresDialect can handle custom types and array types, but not array of custom types. Tthe type support within arrays is limited to what is listed in the catalyst type here: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala#L69-L98.] Since a custom type won't match any of those, `None` is returned and eventually `JdbcUtils` throws this exception:

```

java.sql.SQLException: Unsupported type ARRAY
  at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedJdbcTypeError(QueryExecutionErrors.scala:682)
  at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getCatalystType(JdbcUtils.scala:249)
  at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$getSchema$1(JdbcUtils.scala:327)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:327)

```

The postgresql driver change was part of [Issue#1948|[https://github.com/pgjdbc/pgjdbc/issues/1948].]

 

I did make a change locally and returned `StringType` instead of `None` for the default case, and that worked fine, but I don't know if that's the desired solution or not.

I created a gist with a code snippet to recreate the issue and run it via spark-shell: https://gist.github.com/sdurfey/f9e73cffaeb90cd9c69dcc771fe59f08
Bug: Redundant `LevelDB.get` in `RemoteBlockPushResolver`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Aug/22 08:39
Affects Version/s: 3.4.0

{code:java}
void removeAppAttemptPathInfoFromDB(String appId, int attemptId) {
  AppAttemptId appAttemptId = new AppAttemptId(appId, attemptId);
  if (db != null) {
    try {
      byte[] key = getDbAppAttemptPathsKey(appAttemptId);
      if (db.get(key) != null) {
        db.delete(key);
      }
    } catch (Exception e) {
      logger.error("Failed to remove the application attempt {} local path in DB",
          appAttemptId, e);
    }
  }
}
 {code}
No need to check `db.get(key) != null` before delete. LevelDB will handle this scene.
Bug: Upgrade dropwizard metrics to 4.2.10
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Aug/22 02:53
Affects Version/s: 3.4.0

* [https://github.com/dropwizard/metrics/releases/tag/v4.2.8]
 * [https://github.com/dropwizard/metrics/releases/tag/v4.2.9]
 * https://github.com/dropwizard/metrics/releases/tag/v4.2.10
Bug: Optimize Utils.getIteratorSize for Scala 2.13 refer to IterableOnceOps.size
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Aug/22 02:32
Affects Version/s: 3.4.0

{{Utils.getIteratorSize}} is slightly slower than {{Iterator.size}} when using Scala 2.13, but it seems that we can't call {{Iterator.size}} directly, because {{Utils.getIteratorSize}} method returns {{Long}} type data, using {{Iterator.size}} directly can't ensure that overflow will not occur.
Bug: shift() func need support periods=0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Aug/22 01:50
Affects Version/s: 3.2.2

PySpark raises Error when we call shift func with periods=0.

The behavior of Pandas will return a same copy for the said obj.

 

PySpark:
{code:java}
>>> df = ps.DataFrame({'Col1': [10, 20, 15, 30, 45], 'Col2': [13, 23, 18, 33, 48],'Col3': [17, 27, 22, 37, 52]},columns=['Col1', 'Col2', 'Col3'])
>>> df.Col1.shift(periods=3)
22/08/02 09:37:51 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/08/02 09:37:51 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/08/02 09:37:51 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/08/02 09:37:52 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/08/02 09:37:52 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
0     NaN
1     NaN
2     NaN
3    10.0
4    20.0
Name: Col1, dtype: float64
>>> df.Col1.shift(periods=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/spark/spark/python/pyspark/pandas/base.py", line 1170, in shift
    return self._shift(periods, fill_value).spark.analyzed
  File "/home/spark/spark/python/pyspark/pandas/spark/accessors.py", line 256, in analyzed
    return first_series(DataFrame(self._data._internal.resolved_copy))
  File "/home/spark/spark/python/pyspark/pandas/utils.py", line 589, in wrapped_lazy_property
    setattr(self, attr_name, fn(self))
  File "/home/spark/spark/python/pyspark/pandas/internal.py", line 1173, in resolved_copy
    sdf = self.spark_frame.select(self.spark_columns + list(HIDDEN_COLUMNS))
  File "/home/spark/spark/python/pyspark/sql/dataframe.py", line 2073, in select
    jdf = self._jdf.select(self._jcols(*cols))
  File "/home/spark/.pyenv/versions/3.8.13/lib/python3.8/site-packages/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/home/spark/spark/python/pyspark/sql/utils.py", line 196, in deco
    raise converted from None
pyspark.sql.utils.AnalysisException: Cannot specify window frame for lag function
 {code}
Pandas:
{code:java}
>>> pdf = pd.DataFrame({'Col1': [10, 20, 15, 30, 45], 'Col2': [13, 23, 18, 33, 48],'Col3': [17, 27, 22, 37, 52]},columns=['Col1', 'Col2', 'Col3'])
>>> pdf.Col1.shift(periods=3)
0     NaN
1     NaN
2     NaN
3    10.0
4    20.0
Name: Col1, dtype: float64
>>> pdf.Col1.shift(periods=0)
0    10
1    20
2    15
3    30
4    45
Name: Col1, dtype: int64
 {code}
Bug: Improvement to Substring function
Status: Open | Priority: Minor | Resolution: 
Created: 01/Aug/22 18:52
Affects Version/s: 3.2.2

Hello,

We hit a potential limitation of the substring function today that we think might be useful to contribute back to spark, relating to that fact that [substring |https://github.com/apache/spark/blob/775b5ced70c87d6a8709f44eb4a5de48c286a51d/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L2952] only takes integers for it's pos and len parameters.

Consider the case where you have a list of ID's, and you need to find substrings in these IDs that are dependent on another column (for example, a column expressing the kind of product that the ID represents).

e.g.

{{Table:}}

ID                  | Type
ABCD12345 |   Water
ABCD12        |  Soda

Desired:

ID                  | Type
12345           |   Water
12                  |  Soda

At the moment, you can only extract substrings of fixed position and length- but if IDs have a variable suffix, it may be desirable to run a substring of different lengths based on the 'Type' column (e.g. before running a substring on this 'Type' column, I could run a .replace with a dict of [type]: [substring length], or performed a join to another dataset that contains the substring lengths).

In this case, it's fairly straightforward to solve with a chain of .when statements; but if there are thousands of types, it may not be as easy.

This seems fairly niche, but I'd be happy to give the implementation a go if this is something that the community would find useful/ has been seen before.
Bug: The input parameter of nsmallest should be validated as Integer
Status: In Progress | Priority: Minor | Resolution: 
Created: 02/Aug/22 02:32
Affects Version/s: 3.2.2

The input parameter of nsmallest should be validated as Integer. So I think we might miss this validation.

And PySpark will raise Error when we input the strange types. Such as

 

PySpark:
{code:java}
>>> df = ps.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]}, columns=['A', 'B']) 
>>> df.groupby(['A'])['B'].nsmallest(1)
 A    
1  0    3 
2  1    4 
3  2    5 
4  3    6 
Name: B, dtype: int64
>>> df.groupby(['A'])['B'].nsmallest(True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/spark/spark/python/pyspark/pandas/groupby.py", line 3598, in nsmallest
    sdf.withColumn(temp_rank_column, F.row_number().over(window))
  File "/home/spark/spark/python/pyspark/sql/dataframe.py", line 2129, in filter
    jdf = self._jdf.filter(condition._jc)
  File "/home/spark/.pyenv/versions/3.8.13/lib/python3.8/site-packages/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/home/spark/spark/python/pyspark/sql/utils.py", line 196, in deco
    raise converted from None
pyspark.sql.utils.AnalysisException: cannot resolve '(__rank__ <= true)' due to data type mismatch: differing types in '(__rank__ <= true)' (int and boolean).;
'Filter (__rank__#4995 <= true)
+- Project [__index_level_0__#4988L, __index_level_1__#4989L, B#4979L, __natural_order__#4983L, __rank__#4995]
   +- Project [__index_level_0__#4988L, __index_level_1__#4989L, B#4979L, __natural_order__#4983L, __rank__#4995, __rank__#4995]
      +- Window [row_number() windowspecdefinition(__index_level_0__#4988L, B#4979L ASC NULLS FIRST, __natural_order__#4983L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS __rank__#4995], [__index_level_0__#4988L], [B#4979L ASC NULLS FIRST, __natural_order__#4983L ASC NULLS FIRST]
         +- Project [__index_level_0__#4988L, __index_level_1__#4989L, B#4979L, __natural_order__#4983L]
            +- Project [A#4978L AS __index_level_0__#4988L, __index_level_0__#4977L AS __index_level_1__#4989L, B#4979L, __natural_order__#4983L]
               +- Project [__index_level_0__#4977L, A#4978L, B#4979L, monotonically_increasing_id() AS __natural_order__#4983L]
                  +- LogicalRDD [__index_level_0__#4977L, A#4978L, B#4979L], false


{code}
Bug: window and min_periods parameters in rolling func need to be validated as a Integer
Status: In Progress | Priority: Minor | Resolution: 
Created: 02/Aug/22 02:14
Affects Version/s: 3.2.2

window and min_periods parameters is not be validated in rolling function.

 
{code:java}
>>> s = ps.Series([4, 3, 5, 2, 6])
>>> s.rolling('STRING').sum()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/spark/spark/python/pyspark/pandas/generic.py", line 2707, in rolling
    return Rolling(self, window=window, min_periods=min_periods)
  File "/home/spark/spark/python/pyspark/pandas/window.py", line 179, in __init__
    super().__init__(window, min_periods)
  File "/home/spark/spark/python/pyspark/pandas/window.py", line 147, in __init__
    if window < 0:
TypeError: '<' not supported between instances of 'str' and 'int'
>>> 
 {code}
Bug: The prefix/suffix parameter validation in add_prefix/add_suffix should follow the pandas behavior
Status: In Progress | Priority: Minor | Resolution: 
Created: 02/Aug/22 01:34
Affects Version/s: 3.2.2

We need to follow the pandas behavior of prefix/suffix parameter validation in add_prefix/add_suffix.

Now, we force to validate it as a String type. But pandas looks all values which can be formated as String(implement __str__ func). So it's different here.

PySpark:
{code:java}
>>> from pyspark import pandas as ps
>>> df = ps.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]}, columns=['A', 'B'])
>>> df.add_suffix(666)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/spark/spark/python/pyspark/pandas/frame.py", line 9060, in add_suffix
    assert isinstance(suffix, str)
AssertionError
>>> df.add_suffix(True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/spark/spark/python/pyspark/pandas/frame.py", line 9060, in add_suffix
    assert isinstance(suffix, str)
AssertionError {code}
Pandas:
{code:java}
>>> pd.__version__
'1.3.5'
>>> pdf.add_suffix(0.1)
   A0.1  B0.1
0     1     3
1     2     4
2     3     5
3     4     6 
>>> pdf.add_suffix(True)
   ATrue  BTrue
0      1      3
1      2      4
2      3      5
3      4      6
 {code}
Bug: PySpark: DataFrame.cache docstring gives wrong storage level
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 20/Aug/22 12:26
Affects Version/s: 3.3.0

The docstring of the `DataFrame.cache()` method currently states that it uses a serialized storage level
{code:java}
Persists the :class:`DataFrame` with the default storage level (`MEMORY_AND_DISK`).
[...]
-        The default storage level has changed to `MEMORY_AND_DISK` to match Scala in 2.0.{code}
while `DataFrame.persist()` states that it uses a deserialized storage level
{code:java}
If no storage level is specified defaults to (`MEMORY_AND_DISK_DESER`)
[...]
The default storage level has changed to `MEMORY_AND_DISK_DESER` to match Scala in 3.0.{code}
 

However, in practice both `.cache()` and `.persist()` use deserialized storage levels:
{code:java}
import pyspark
from pyspark.sql import SparkSession
from pyspark import StorageLevel
print(pyspark.__version__)
# 3.3.0

spark = SparkSession.builder.master("local[2]").getOrCreate()

df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a", "col_b"])
df = df.cache()
df.count()
# Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"

df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a", "col_b"])
df = df.persist()
df.count()
# Storage level in Spark UI: "Disk Memory Deserialized 1x Replicated"

df = spark.createDataFrame(zip(["A"] * 1000, ["B"] * 1000), ["col_a", "col_b"])
df = df.persist(StorageLevel.MEMORY_AND_DISK)
df.count()
# Storage level in Spark UI: "Disk Memory Serialized 1x Replicated"{code}
Bug: Skip PVC cleanup when driver doesn't own PVCs
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 03/Aug/22 05:00
Affects Version/s: 3.3.0

From Spark32 . as a part of [https://github.com/apache/spark/pull/32288] , functionality is added to delete PVC if the Spark driver died. 

[https://github.com/apache/spark/blob/786a70e710369b195d7c117b33fe9983044014d6/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L144]

 

However there are cases , where spark on K8s doesn't use PVC and use host path for storage. 

[https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes]

 

Now  in those cases ,
 * it request to delete PVC (which is not required) .
 * It also tries to delete in the case where driver doesn't own the PV (or spark.kubernetes.driver.ownPersistentVolumeClaim is false) 
 * Moreover in the cluster , where Spark user doesn't have access to list or delete PVC , it throws exception .  

 

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: [https://kubernetes.default.svc/api/v1/namespaces/<>/persistentvolumeclaims?labelSelector=spark-app-selector%3Dspark-332bd09284b3442f8a6a214fabcd6ab1|https://kubernetes.default.svc/api/v1/namespaces/dpi-dev/persistentvolumeclaims?labelSelector=spark-app-selector%3Dspark-332bd09284b3442f8a6a214fabcd6ab1]. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. persistentvolumeclaims is forbidden: User "system:serviceaccount:dpi-dev:spark" cannot list resource "persistentvolumeclaims" in API group "" in the namespace "<>".

 

*Solution*

Ideally there should be configuration spark.kubernetes.driver.pvc.deleteOnTermination or use spark.kubernetes.driver.ownPersistentVolumeClaim  which should be checked before calling to delete PVC. If user have not set up PV or if the driver doesn't own  then there is no need to call the api and delete PVC .
Bug: Update test plugins to latest versions
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Aug/22 11:02
Affects Version/s: 3.4.0

Include:
 * 1.scalacheck (from 1.16.0 to 1.17.0)
 * 2.maven-surefire-plugin (from 3.0.0-M5 to 3.0.0-M7)
 * 3.maven-dependency-plugin (from 3.1.1 to 3.3.0)
Bug: [SPARK][SQL] feat: SparkSession.confing(Map)
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 21/Aug/22 08:45
Affects Version/s: 3.3.0

[https://github.com/apache/spark/pull/37478] 

- as-is
{code:java}
private fun config(builder: SparkSession.Builder): SparkSession.Builder {
        val map = YamlUtils.read(this::class.java, "spark", Extension.YAML)

        var b = builder
        map.keys.forEach {
            val k = it
            val v = map[k]

            b = when (v) {
                is Long -> b.config(k, v)
                is String -> b.config(k, v)
                is Double -> b.config(k, v)
                is Boolean -> b.config(k, v)
                else -> b
            }
        }

        return b
    }
} {code}
- to-be
{code:java}
private fun config(builder: SparkSession.Builder): SparkSession.Builder {
        val map = YamlUtils.read(this::class.java, "spark", Extension.YAML)

        return b.config(map)
    }
} {code}
Bug: Can't set pyspark log level in log4j2.properties
Status: Resolved | Priority: Trivial | Resolution: Invalid
Created: 06/Aug/22 16:43
Affects Version/s: 3.3.0

In previous versions of Apache Spark it was possible to configure pyspark log level by adding following line to log4j.properties:

 

{{log4j.logger.org.apache.spark.api.python.PythonGatewayServer=ERROR}}

 

With the move to log4j2.properties format, this no longer works. I have tried modifying the above line to:

{{logger.pyspark.name = org.apache.spark.api.python.PythonGatewayServer}}
{{logger.pyspark.level = error}}

 

but that does not work either. 

I would appreciate any help with the correct syntax.
Bug: Cast in having aggregate expressions returns the wrong result
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 22/Apr/20 09:56
Affects Version/s: 2.2.3

Cast in having aggregate expressions returns the wrong result.

See the below tests: 
{code:java}

scala> spark.sql("create temp view t(a, b) as values (1,10), (2, 20)")
res0: org.apache.spark.sql.DataFrame = []

scala> val query = """
     | select sum(a) as b, '2020-01-01' as fake
     | from t
     | group by b
     | having b > 10;"""

scala> spark.sql(query).show()
+---+----------+
|  b|      fake|
+---+----------+
|  2|2020-01-01|
+---+----------+

scala> val query = """
     | select sum(a) as b, cast('2020-01-01' as date) as fake
     | from t
     | group by b
     | having b > 10;"""

scala> spark.sql(query).show()
+---+----+
|  b|fake|
+---+----+
+---+----+
{code}
The SQL parser in Spark creates Filter(..., Aggregate(...)) for the HAVING query, and Spark has a special analyzer rule ResolveAggregateFunctions to resolve the aggregate functions and grouping columns in the Filter operator.
 
It works for simple cases in a very tricky way as it relies on rule execution order:
1. Rule ResolveReferences hits the Aggregate operator and resolves attributes inside aggregate functions, but the function itself is still unresolved as it's an UnresolvedFunction. This stops resolving the Filter operator as the child Aggrege operator is still unresolved.
2. Rule ResolveFunctions resolves UnresolvedFunction. This makes the Aggrege operator resolved.
3. Rule ResolveAggregateFunctions resolves the Filter operator if its child is a resolved Aggregate. This rule can correctly resolve the grouping columns.
 
In the example query, I put a CAST, which needs to be resolved by rule ResolveTimeZone, which runs after ResolveAggregateFunctions. This breaks step 3 as the Aggregate operator is unresolved at that time. Then the analyzer starts next round and the Filter operator is resolved by ResolveReferences, which wrongly resolves the grouping columns.
Bug: file source backward compatibility after calendar switch
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 10/Apr/20 03:33
Affects Version/s: 3.0.0

In Spark 3.0, we switch to the Proleptic Gregorian calendar by using the Java 8 datetime APIs. This makes Spark follow the ISO and SQL standard, but introduces some backward compatibility problems:
1. may read wrong data from the data files written by Spark 2.4
2. may have perf regression
Bug: Fix nested CTE substitution
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Apr/20 18:40
Affects Version/s: 3.0.0

The following nested CTE should result empty result instead of {{1}}

{noformat}
WITH t(c) AS (SELECT 1)
SELECT * FROM t
WHERE c IN (
  WITH t(c) AS (SELECT 2)
  SELECT * FROM t
)
{noformat}
Bug: Accessing the sequence number and partition id for records in Kinesis adapter
Status: Resolved | Priority: Critical | Resolution: Invalid
Created: 10/Apr/20 09:25
Affects Version/s: 2.4.3

We are using spark 2.4.3 in java. We would like to log the partition key and the sequence number of every event. The overloaded create stream function of the kinesis utils always throws a compilation error.

 

Function<Record,Record> printSeq = s -> s;
 KinesisUtils.createStream(
 jssc,
 appName,
 streamName,
 endPointUrl,
 regionName,
 InitialPositionInStream.TRIM_HORIZON,
 kinesisCheckpointInterval,
 StorageLevel.MEMORY_AND_DISK_SER(),
 printSeq,
 Record.class);

*{color:#172b4d}Record is of type com.amazonaws.services.kinesis.model.Record{color}*

 

The exception is as follows:
{quote}no suitable method found for createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class)
{quote}
JAVA DOCS : [https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-]

Is there a way out?
Bug: Pyspark SparkContext.cancelJobGroup do not work correctly
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 24/Apr/20 03:22
Affects Version/s: 2.4.5

Pyspark SparkContext.cancelJobGroup do not work correctly. This is an issue existing for a long time. This is because of pyspark thread didn't pinned to jvm thread when invoking java side methods, which leads to all pyspark API which related to java local thread variables do not work correctly. (Including `sc.setLocalProperty`, `sc.cancelJobGroup`, `sc.setJobDescription` and so on.)

This is serious issue. Now there's an experimental pyspark 'PIN_THREAD' mode added in spark-3.0 which address it, but the 'PIN_THREAD' mode exists two issue:
* It is disabled by default. We need to set additional environment variable to enable it.
* There's memory leak issue which haven't been addressed.

Now there's a series of project like hyperopt-spark, spark-joblib which rely on `sc.cancelJobGroup` API (use it to stop running jobs in their code). So it is critical to address this issue and we hope it work under default pyspark mode. An optional approach is implementing methods like `rdd.setGroupAndCollect`.