Repository: mars-project/mars

Issue:
[BUG] race condition: duplicate decref of subtask input chunk
<!--
Thank you for your contribution!

Please review https://github.com/mars-project/mars/blob/master/CONTRIBUTING.rst before opening an issue.
-->

**Describe the bug**
Let us suppose stage A have two Subtask S1 and S2, and they have the same input chunk C

1. S1 got an error, and stage_processor.done has been set.
2. S2 call set_subtask_result, it already reduces reference count of C  but not set `stage_processor.results[C.key]`
3. `TaskProcessorActor` find `stage_processor` got an error and call `self._cur_processor.incref_stage(stage_processor)` in function `TaskProcessorActor.start`, it will also reduce the reference count of C which is input of S2.

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: python 3.7.9
2. The version of Mars you use: master
3. Minimized code to reproduce the error:

you can run this bash: `pytest mars/services/task/supervisor/tests/test_task_manager.py::test_error_task -s` in this branch: https://github.com/Catch-Bull/mars/tree/race_condition_case, which base latest mars master,and add some sleep to make reproducing easily

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/mars/services/task/supervisor/processor.py b/mars/services/task/supervisor/processor.py
--- a/mars/services/task/supervisor/processor.py
+++ b/mars/services/task/supervisor/processor.py
@@ -215,8 +215,9 @@ def _get_decref_stage_chunk_keys(
             if error_or_cancelled:
                 # error or cancel, rollback incref for subtask results
                 for subtask in subtask_graph:
-                    if stage_processor.subtask_results.get(subtask):
+                    if subtask.subtask_id in stage_processor.decref_subtask:
                         continue
+                    stage_processor.decref_subtask.add(subtask.subtask_id)
                     # if subtask not executed, rollback incref of predecessors
                     for inp_subtask in subtask_graph.predecessors(subtask):
                         for result_chunk in inp_subtask.chunk_graph.results:
@@ -821,9 +822,12 @@ async def set_subtask_result(self, subtask_result: SubtaskResult):
                 # Since every worker will call supervisor to set subtask result,
                 # we need to release actor lock to make `decref_chunks` parallel to avoid blocking
                 # other `set_subtask_result` calls.
-                yield self._decref_input_subtasks(
-                    subtask, stage_processor.subtask_graph
-                )
+                if subtask.subtask_id not in stage_processor.decref_subtask:
+                    stage_processor.decref_subtask.add(subtask.subtask_id)
+                    yield self._decref_input_subtasks(
+                        subtask, stage_processor.subtask_graph
+                    )
+
             except:  # noqa: E722  # nosec  # pylint: disable=bare-except  # pragma: no cover
                 _, err, tb = sys.exc_info()
                 if subtask_result.status not in (
diff --git a/mars/services/task/supervisor/stage.py b/mars/services/task/supervisor/stage.py
--- a/mars/services/task/supervisor/stage.py
+++ b/mars/services/task/supervisor/stage.py
@@ -67,6 +67,9 @@ def __init__(
         self.subtask_results: Dict[Subtask, SubtaskResult] = dict()
         self._submitted_subtask_ids = set()
 
+        # All subtask IDs whose input chunk reference count is reduced.
+        self.decref_subtask = set()
+
         self._band_manager: Dict[BandType, mo.ActorRef] = dict()
 
         # result


Test patch: