av9ash/GitBugs huggingface dataset:
Bug: Log ExecutorDecommission as INFO level in TaskSchedulerImpl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/22 01:31
Affects Version/s: 3.4.0

Like the other module, `TaskSchedulerImpl` should log the decommission as `INFO` level.
{code}
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission executors: 8
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 to decommissioning.
22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: Executor decommission.
22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason statistics: ...
22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(8, 100.103.40.13, 43353, None)
22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in removeExecutor
22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 (epoch 7)
22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
22/05/28 01:25:34 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 8
{code}
Bug: Respect ps.concat sort parameter to follow pandas behavior
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/22 08:13
Affects Version/s: 3.4.0

[https://github.com/Yikun/spark/pull/101/checks?check_run_id=6621103945]

 

[https://github.com/pandas-dev/pandas/issues/47127]
Bug: '_SubTest' object has no attribute 'elapsed_time'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/22 03:11
Affects Version/s: 3.4.0

{code:java}
Traceback (most recent call last):
1582  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
1583    return _run_code(code, main_globals, None,
1584  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
1585    exec(code, run_globals)
1586  File "/__w/spark/spark/python/pyspark/pandas/tests/test_namespace.py", line 585, in <module>
1587    unittest.main(testRunner=testRunner, verbosity=2)
1588  File "/usr/lib/python3.9/unittest/main.py", line 101, in __init__
1589    self.runTests()
1590  File "/usr/lib/python3.9/unittest/main.py", line 271, in runTests
1591    self.result = testRunner.run(self.test)
1592  File "/usr/local/lib/python3.9/dist-packages/xmlrunner/xmlrunner.py", line 421, in run
1593    result.printErrors()
1594  File "/usr/lib/python3.9/unittest/runner.py", line 110, in printErrors
1595    self.printErrorList('FAIL', self.failures)
1596  File "/usr/local/lib/python3.9/dist-packages/xmlrunner/xmlrunner.py", line 217, in printErrorList
1597    '%s [%.3fs]: %s' % (flavour, test_info.elapsed_time,
1598AttributeError: '_SubTest' object has no attribute 'elapsed_time' {code}
[https://github.com/xmlrunner/unittest-xml-reporting/issues/218#issuecomment-665088941]

 

Looks like we need to switch xmlrunner to https://pypi.org/project/unittest-xml-reporting/
Bug: groupby.apply doc test failed when SPARK_CONF_ARROW_ENABLED disable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/22 08:31
Affects Version/s: 3.4.0

{code:java}
Traceback (most recent call last):
  File "/Users/yikun/venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3361, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-3cad40f83c12>", line 1, in <cell line: 1>
    g.apply(plus_min).sort_index()
  File "/Users/yikun/spark/python/pyspark/pandas/groupby.py", line 1444, in apply
    psser_or_psdf = ps.from_pandas(pser_or_pdf)
  File "/Users/yikun/spark/python/pyspark/pandas/namespace.py", line 153, in from_pandas
    return DataFrame(pobj)
  File "/Users/yikun/spark/python/pyspark/pandas/frame.py", line 457, in __init__
    internal = InternalFrame.from_pandas(pdf)
  File "/Users/yikun/spark/python/pyspark/pandas/internal.py", line 1473, in from_pandas
    sdf = default_session().createDataFrame(pdf, schema=schema)
  File "/Users/yikun/spark/python/pyspark/sql/session.py", line 961, in createDataFrame
    return super(SparkSession, self).createDataFrame(  # type: ignore[call-overload]
  File "/Users/yikun/spark/python/pyspark/sql/pandas/conversion.py", line 437, in createDataFrame
    return self._create_dataframe(converted_data, schema, samplingRatio, verifySchema)
  File "/Users/yikun/spark/python/pyspark/sql/session.py", line 1006, in _create_dataframe
    rdd, struct = self._createFromLocal(map(prepare, data), schema)
  File "/Users/yikun/spark/python/pyspark/sql/session.py", line 698, in _createFromLocal
    data = list(data)
  File "/Users/yikun/spark/python/pyspark/sql/session.py", line 980, in prepare
    verify_func(obj)
  File "/Users/yikun/spark/python/pyspark/sql/types.py", line 1763, in verify
    verify_value(obj)
  File "/Users/yikun/spark/python/pyspark/sql/types.py", line 1741, in verify_struct
    verifier(v)
  File "/Users/yikun/spark/python/pyspark/sql/types.py", line 1763, in verify
    verify_value(obj)
  File "/Users/yikun/spark/python/pyspark/sql/types.py", line 1686, in verify_long
    verify_acceptable_types(obj)
  File "/Users/yikun/spark/python/pyspark/sql/types.py", line 1633, in verify_acceptable_types
    raise TypeError(
TypeError: field B: LongType() can not accept object 2 in type <class 'numpy.int64'> {code}
Bug: timestamp type support precision, example: timestamp(3)
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 26/May/22 11:27
Affects Version/s: 3.4.0

Dremio supports syntax: select CAST(-123456789 AS TIMESTAMP(3)) as times，spark sql cannot specify 3

Dremio uses calcite，There is an incompatibility problem when converting dremio sql to spark sql
Bug: Correct the behavior of creating DataFrame from an RDD
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/May/22 18:45
Affects Version/s: 3.4.0

Correct the behavior of creating DataFrame from an RDD **with `0` or an empty list as the first element**.

 

Before:
```py
>>> spark.createDataFrame(spark._sc.parallelize([0, 1]))
Traceback (most recent call last):
...
ValueError: The first row in RDD is empty, can not infer schema

>>> spark.createDataFrame(spark._sc.parallelize([[], []]))
Traceback (most recent call last):
...
ValueError: The first row in RDD is empty, can not infer schema
```

After:
```py
>>> spark.createDataFrame(spark._sc.parallelize([0, 1]))
Traceback (most recent call last):                                              
...
TypeError: Can not infer schema for type: <class 'int'>

>>> spark.createDataFrame(spark._sc.parallelize([[], []]))
DataFrame[]                                                                     
>>> spark.createDataFrame(spark._sc.parallelize([[], []])).show()
++
||
++
||
||
++
```
Bug: Question of job division in "df.groupBy().applyInPandas"
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 25/May/22 13:16
Affects Version/s: 3.2.1

My program runs in spark on Yarn environment and has four nodes in total. My num executors is set to 4, and so is the executor cores. When I use "df.groupBy().applyInPandas" to process my dataframe, the program is always divided into two jobs. The first job contains only one task, and the second job contains three tasks. And the two jobs have the same DAG diagram and perform exactly the same operations, but the data is different. As a result, the execution time of my program is about doubled. It should be that only one job contains four tasks. This problem has bothered me for a long time and I can't find the reason. This code is a simple example I use to test. My dataframe stores the information of the image data saved by HDFS, including three columns: "Id", "path" and "category". "Id" indicates the node on which the image is located, "path" indicates the specific path, "category" indicates the image category.  Job1 and job2 in "WebUI.png" correspond to the two divided jobs. "job1". "Job1.png" and "job2. png" show the executor and node information of the two jobs.
{code:java}
import os
import pandas as pd
from pyspark.sql import SparkSession
from pyhdfs import HdfsClient

spark = SparkSession.builder.appName("test"). \
    config("spark.sql.shuffle.partitions", "40"). \
    config("spark.default.parallelism", "40"). \
    config("spark.sql.execution.arrow.pyspark.enabled", "true"). \
    getOrCreate()


def process(key, paths):
    a = len(paths.path)
    for i in range(100000000):
        a+=1
    a = str(a)
    return pd.DataFrame([key+(a,)])


if __name__ == '__main__':
    sc = spark.sparkContext
    client = HdfsClient(hosts="master:9870", user_name="hadoop")
    dataset_dir = "/Datasets"
    files_pd = pd.DataFrame()
    for slave, per_dataset_dir in enumerate(client.listdir(dataset_dir)):
        child_path = os.path.join(dataset_dir, per_dataset_dir)
        files = pd.DataFrame([
            [slave, os.path.join(str(child_path), str(child_dir_name), str(filename)), index]
            for index, child_dir_name in enumerate(client.listdir(child_path))
            for filename in client.listdir(os.path.join(child_path, child_dir_name))])
        files_pd = pd.concat([files_pd, files])
    files_pd = files_pd.sample(frac=1).reset_index(drop=True)
    spark_files = spark.createDataFrame(files_pd, ("id", "path", "category"))
    result = spark_files.groupby("id").applyInPandas(process, schema="id long, path string")
    result.show()

 {code}
Bug: Improve PySpark API reference to be more readable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/May/22 07:40
Affects Version/s: 3.3.0

Currently, the PySpark documentation especially ["Spark SQL" part|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql.html#] is not-well organized so it's a bit uncomfortable to be read.

For example, the [pyspark.sql.SparkSession|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.SparkSession.html] class shows every available method its sub-contents, but pyspark.sql.DataFrameReader doesn't have it's own top-level class page, and also its available methods.

So we might need to refine the document to make it more readable so that users can easily find the methods they want.
Bug: Flaky Test: pyspark.sql.tests.test_dataframe.DataFrameTests test_df_is_empty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/May/22 03:28
Affects Version/s: 3.1.3

{{test_df_is_empty}} is flaky. For example, a recent PR: https://github.com/apache/spark/pull/36580
https://github.com/panbingkun/spark/runs/6525997469?check_suite_focus=true

Possibly introduced from SPARK-39084

{code}
test_df_is_empty (pyspark.sql.tests.test_dataframe.DataFrameTests) ... 
[Stage 6:>                                                          (0 + 1) / 1]

                                                                                
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fd84a1486ff, pid=4021, tid=0x00007fd8016a2700
#
# JRE version: OpenJDK Runtime Environment (Zulu 8.62.0.19-CA-linux64) (8.0_332-b09) (build 1.8.0_332-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.332-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 9116 C2 org.apache.spark.unsafe.UnsafeAlignedOffset.getSize(Ljava/lang/Object;J)I (51 bytes) @ 0x00007fd84a1486ff [0x00007fd84a1486e0+0x1f]
#
# Core dump written. Default location: /__w/spark/spark/core or core.4021
#
# An error report file with more information is saved as:
# /__w/spark/spark/hs_err_pid4021.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 36358)
Traceback (most recent call last):
  File "/usr/lib/pypy3/lib-python/3/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/pypy3/lib-python/3/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/pypy3/lib-python/3/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/pypy3/lib-python/3/socketserver.py", line 720, in __init__
    self.handle()
  File "/__w/spark/spark/python/pyspark/accumulators.py", line 281, in handle
    poll(accum_updates)
  File "/__w/spark/spark/python/pyspark/accumulators.py", line 253, in poll
    if func():
  File "/__w/spark/spark/python/pyspark/accumulators.py", line 257, in accum_updates
    num_updates = read_int(self.rfile)
  File "/__w/spark/spark/python/pyspark/serializers.py", line 595, in read_int
    raise EOFError
{code}
Bug: Make PandasOnSparkTestCase inherit ReusedSQLTestCase
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/22 11:09
Affects Version/s: 3.4.0

PandasOnSparkTestCase has some legacy codes e.g., not stopping in stop {{tearDownClass}}. We don't need such logic anymore. That logic was when the codes are in Koalas repo that runs the tests sequentially.
Bug: Python foreachBatch streaming query cannot be stopped gracefully after pin thread mode is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 06:18
Affects Version/s: 3.0.3

For example,

{code}
import time

def func(batch_df, batch_id):
    time.sleep(10)
    print(batch_df.count())

q = spark.readStream.format("rate").load().writeStream.foreachBatch(func).start()
time.sleep(5)
q.stop()
{code}

works find with pinned thread mode is disabled. Whe pinned thread mode is enabled:

{code}
22/05/18 15:23:24 ERROR MicroBatchExecution: Query [id = 2538f8a2-c6e4-44c9-bf38-e6dab555267e, runId = 1d500478-1d77-46aa-b35a-585264a809b9] terminated with error
py4j.Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last):
  File "/.../spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 617, in _call_proxy
    return_value = getattr(self.pool[obj_id], method)(*params)
  File "/.../spark/python/pyspark/sql/utils.py", line 272, in call
    raise e
  File "/.../spark/python/pyspark/sql/utils.py", line 269, in call
    self.func(DataFrame(jdf, self.session), batch_id)
  File "<stdin>", line 3, in func
  File "/.../spark/python/pyspark/sql/dataframe.py", line 804, in count
    return int(self._jdf.count())
  File "/.../spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/.../spark/python/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/.../spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o44.count.
: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
	at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:242)
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:258)
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:187)
	at org.apache.spark.util.ThreadUtils$.awaitReady(ThreadUtils.scala:334)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:943)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2227)
{code}
Bug: codegen cause NullPointException
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 18/May/22 07:37
Affects Version/s: 2.3.0

The following code raises NullPointException
{code:sql}
SELECT
  fk4c7a8cfc,
  fka54f2a73,
  fk37e266f7
FROM
  be2a04fad4a24848bee641825e5b3466
WHERE
  (
    fk4c7a8cfc is not null
    and fk4c7a8cfc<> ''
  )
LIMIT
  1000

{code}
However, if so, it is normal
{code:sql}
SELECT
  fk4c7a8cfc,
  fka54f2a73,
  fk37e266f7
FROM
  be2a04fad4a24848bee641825e5b3466
WHERE
  (
    fk4c7a8cfc is not null
    and '' <> fk4c7a8cfc
  )
LIMIT
  1000

{code}
I just put the '' in where in front.

The reason for this problem is that the data contains null values.

*_org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext#genEqual_*
{code:scala}
  def genEqual(dataType: DataType, c1: String, c2: String): String = dataType match {
    case BinaryType => s"java.util.Arrays.equals($c1, $c2)"
    case FloatType =>
      s"((java.lang.Float.isNaN($c1) && java.lang.Float.isNaN($c2)) || $c1 == $c2)"
    case DoubleType =>
      s"((java.lang.Double.isNaN($c1) && java.lang.Double.isNaN($c2)) || $c1 == $c2)"
    case dt: DataType if isPrimitiveType(dt) => s"$c1 == $c2"
    case dt: DataType if dt.isInstanceOf[AtomicType] => s"$c1.equals($c2)"
    case array: ArrayType => genComp(array, c1, c2) + " == 0"
    case struct: StructType => genComp(struct, c1, c2) + " == 0"
    case udt: UserDefinedType[_] => genEqual(udt.sqlType, c1, c2)
    case NullType => "false"
    case _ =>
      throw new IllegalArgumentException(
        "cannot generate equality code for un-comparable type: " + dataType.catalogString)
  }
{code}
{code:scala}
case dt: DataType if dt.isInstanceOf[AtomicType] => s"$c1.equals($c2)"
{code}
Missing null value judgment?
Bug: Consider all values in a python list when inferring schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/22 17:53
Affects Version/s: 3.2.1

Schema inference fails on the following case:
{code:python}
>>> data = [{"a": [1, None], "b": [None, 2]}]
>>> spark.createDataFrame(data)
ValueError: Some of types cannot be determined after inferring
{code}
This is because only the first value in the array is used to infer the element type for the array: [https://github.com/apache/spark/blob/b63674ea5f746306a96ab8c39c23a230a6cb9566/python/pyspark/sql/types.py#L1260]. The element type of the "b" array is inferred as {{NullType}} but I think it makes sense to infer the element type as {{{}LongType{}}}.

One approach to address the above would be to infer the type from the first non-null value in the array. However, consider a case with structs:
{code:python}
>>> spark.conf.set("spark.sql.pyspark.inferNestedDictAsStruct.enabled",  True)
>>> data = [{"a": [{"b": 1}, {"c": 2}]}]
>>> spark.createDataFrame(data).schema
StructType([StructField('a', ArrayType(StructType([StructField('b', LongType(), True)]), True), True)])
{code}
The element type of the "a" array is inferred as a struct with one field, "b". However, it would be convenient to infer the element type as a struct with both fields "b" and "c". Omitted fields from each dictionary would become null values in each struct:
{code:java}
+----------------------+
|                     a|
+----------------------+
|[{1, null}, {null, 2}]|
+----------------------+
{code}
To support both of these cases, the type of each array element could be inferred, and those types could be merged, similar to the approach [here|https://github.com/apache/spark/blob/b63674ea5f746306a96ab8c39c23a230a6cb9566/python/pyspark/sql/session.py#L574-L576].
Bug: Reduce Py4J calls in pyspark.sql.utils.is_timestamp_ntz_preferred
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 01:12
Affects Version/s: 3.3.0

Here https://github.com/apache/spark/blob/master/python/pyspark/sql/utils.py#L296-L302

It unnecessarily accesses to JVM too often. We can just have a single method to avoid that.
Bug: Levearge LocalRelation in createDataFrame with Arrow optimization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/22 05:08
Affects Version/s: 3.4.0

Currently, we use {{LogicalRDD}} that always creates an RDD. in Spark SQL, we have some nice optimization with {{LocalRelation}}. We should leverage this in {{createDataFrame}} in PySpark with Arrow optimization to boost the speed up.
Bug: Access to JVM through passed-in GatewayClient during type conversion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/22 00:47
Affects Version/s: 3.4.0

Access to JVM through passed-in GatewayClient during type conversion.

In customized type converters, we may utilize `jvm` field of passed-in GatewayClient directly to access JVM, rather than rely on the `SparkContext._jvm`.

That's [how](https://github.com/py4j/py4j/blob/master/py4j-python/src/py4j/java_collections.py#L508) Py4J explicit converters access JVM.
Bug: Remove `# doctest: +SKIP` of SPARK-38947/SPARK-39326 when infra dump pandas to 1.4+
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 11/May/22 10:27
Affects Version/s: 3.4.0

[https://github.com/apache/spark/blob/fe85d7912f86c3e337aa93b23bfa7e7e01c0a32e/python/pyspark/pandas/groupby.py#L2333]

[https://github.com/apache/spark/blob/fe85d7912f86c3e337aa93b23bfa7e7e01c0a32e/python/pyspark/pandas/groupby.py#L2265]

all doctest in https://github.com/apache/spark/pull/36712
Bug: ImportError when creating pyspark.pandas document "Supported APIs" if pandas version is low.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/22 02:18
Affects Version/s: 3.4.0

The pyspark.pandas documentation "Supported APIs" will be auto-generated. ([SPARK-38961|https://issues.apache.org/jira/browse/SPARK-38961])

At this point, we need to verify the version of pandas. It can be applied after the docker image used in github action is upgraded and republished at https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage.

Related: https://github.com/apache/spark/pull/36509
Bug: df.rdd.isEmpty() results in unexpected executor failure and JVM crash
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/22 05:38
Affects Version/s: 3.2.0

It was discovered that a particular data distribution in a DataFrame with groupBy clause could result in a JVM crash when calling {{{}df.rdd.isEmpty{}}}.

For example,
{code:java}
data = []
for t in range(0, 10000):
    id = str(uuid.uuid4())
    if t == 0:
        for i in range(0, 99):
            data.append((id,))
    elif t < 10:
        for i in range(0, 75):
            data.append((id,))
    elif t < 100:
        for i in range(0, 50):
            data.append((id,))
    elif t < 1000:
        for i in range(0, 25):
            data.append((id,))
    else:
        for i in range(0, 10):
            data.append((id,))

df = self.spark.createDataFrame(data, ["col"])
df.coalesce(1).write.parquet(tmpPath)

res = self.spark.read.parquet(tmpPath).groupBy("col").count()
print(res.rdd.isEmpty()) # crashes JVM{code}
Reproducible 100% on this dataset.

The ticket is related to (can be thought of as a follow-up for) https://issues.apache.org/jira/browse/SPARK-33277. We need to patch one more place to make sure Python iterator is in sync with Java iterator and is terminated whenever the task is marked as completed.

Note that all other operations appear to work fine: {{{}count{}}}, {{{}collect{}}}.