Repository: PrefectHQ/prefect

Issue:
Setting mapped task runs to a Pending state doesn't always run them
## Description
<!-- A clear description of the bug -->
Setting mapped children to a `Pending` state after a run has failed doesn't re-submit them for execution and their states aren't counted against the flow run's final state.



## Expected Behavior
<!-- What did you expect to happen instead? -->
All pending runs are re-submitted for execution when a flow run is in a non-`Finished` state.



## Reproduction
<!-- A minimal example that exhibits the behavior. -->
This flow will randomly fail (can adjust the incidence in the `Node` class):
```
import prefect
from prefect import Flow, Task
import time
import random


class Version(Task):
    def run(self):
        self.logger.info(f"Running on Prefect v{prefect.__version__}")
        return


class Root(Task):
    def run(self):
        self.logger.info("Root running...")
        time.sleep(random.randint(1, 5))
        self.logger.info("Root complete.")
        return list(range(5))


class Node(Task):
    def run(self):
        self.logger.info(f"{self.name} running...")
        time.sleep(random.randint(1, 5))
        if random.random() > 0.98:
            raise ValueError(f"{self.name} failed :(")
        else:
            self.logger.info(f"{self.name} complete.")
            return list(range(15))

with Flow("Varying Durations") as flow:
    version = Version()

    root = Root()(upstream_tasks=[version])
    node1_1 = Node(name="Node 1_1").map(upstream_tasks=[root])
    node1_2 = Node(name="Node 1_2").map(upstream_tasks=[root])
    node2_1 = Node(name="Node 2_1").map(upstream_tasks=[node1_1, node1_2])
    node3_1 = Node(name="Node 3_1").map(upstream_tasks=[node2_1])
    node3_2 = Node(name="Node 3_2").map(upstream_tasks=[node2_1])
    node4_1 = Node(name="Node 4_1").map(upstream_tasks=[node3_1, node3_2])
```
when downstream mapped runs are "Restarted", meaning the failed runs are put in a `Pending` state and the flow run is re-scheduled, they're never run, leaving a run schematic like this (note the 2 pending runs):
![Screen Shot 2020-09-01 at 8 59 17 PM](https://user-images.githubusercontent.com/27291717/91920131-277d6400-ec96-11ea-9e2c-7285d4d97386.png)



## Environment
<!-- Any additional information about your environment
Optionally run `prefect diagnostics` from the command line and paste the information here. -->
Core version: `0.13.2`
Running on Prefect Cloud
Docker Storage (base image: `python:3.8`)


Discussion:
cc @cicdw 
A couple examples of this for the Prefect team:
https://cloud.prefect.io/prefect/flow-run/e78ab09a-a475-42fe-9267-d3d590fc4c2c
https://cloud.prefect.io/prefect/flow-run/6c6608cd-aeb7-4ce7-b35c-d4588fb79513
Hey @znicholasbrown - I finally figured this one out: your Flow / Tasks don't store their results (because there is no configured result location), and therefore on reruns your Runners don't know how wide the mapped pipelines are (and so all children are effectively "hidden" from view).

I think I can use `n_map_states` (available and populated on newer versions of Core) to ensure all child tasks are actually visited.

Resolving patch:
diff --git a/src/prefect/engine/task_runner.py b/src/prefect/engine/task_runner.py
--- a/src/prefect/engine/task_runner.py
+++ b/src/prefect/engine/task_runner.py
@@ -423,6 +423,8 @@ def check_task_ready_to_map(
             - ENDRUN: either way, we dont continue past this point
         """
         if state.is_mapped():
+            if len(state.map_states) == 0 and state.n_map_states > 0:  # type: ignore
+                state.map_states = [None] * state.n_map_states  # type: ignore
             raise ENDRUN(state)
 
         # we can't map if there are no success states with iterables upstream
@@ -537,7 +539,7 @@ def check_task_is_ready(self, state: State) -> State:
             return state
 
         # the task is mapped, in which case we still proceed so that the children tasks
-        # are generated (note that if the children tasks)
+        # are generated
         elif state.is_mapped():
             self.logger.debug(
                 "Task '%s': task is mapped, but run will proceed so children are generated.",


Test patch:
diff --git a/tests/engine/test_task_runner.py b/tests/engine/test_task_runner.py
--- a/tests/engine/test_task_runner.py
+++ b/tests/engine/test_task_runner.py
@@ -1940,6 +1940,17 @@ def test_mapped_tasks_parents_and_children_respond_to_individual_triggers():
     assert state.is_mapped()
 
 
+def test_mapped_tasks_parent_regenerates_child_pipeline():
+    runner = TaskRunner(task=Task())
+    state = runner.run(
+        upstream_states={Edge(Task(), Task(), mapped=True): Success()},
+        is_mapped_parent=True,
+        state=Mapped(n_map_states=10),
+    )
+    assert state.is_mapped()
+    assert len(state.map_states) == 10
+
+
 def test_retry_has_updated_metadata():
     a, b = Success(result=15), Success(result="abc")