Repository: PrefectHQ/prefect

Issue:
Bug with apply_map Causing Cyclical DAGs
## Opened from the [Prefect Public Slack Community](https://prefect.io/slack)

**fabrice**: Hi,

Why does the result (a tuple) of a mapped function used as input to another mapped function throw `ValueError("Cycle found; flows must be acyclic!")` ? Or to rephrase the question: "Why is it cyclic at all?"

Minimum example:


```
from prefect import Flow, apply_map, task


@task()
def do_something_one(item):
    return item + 1


@task()
def do_something_two(item, item_one):
    return item + item_one


@task()
def do_something_three(item_combination):
    return item_combination[0] + item_combination[1]


# mapping function one
def do_something_with_one_two(item):
    result_one = do_something_one(item)
    result_two = do_something_two(item, result_one)
    return (result_one, result_two)


# mapping function two
def do_something_with_three(item):
    return do_something_three(item)


with Flow("test flow") as flow:
    l = [1, 2, 3]
    results = apply_map(do_something_with_one_two, l)
    results_results = apply_map(do_something_with_three, results)

flow.run()
```

The visualization can be seen below.

**fabrice**: 
```
from prefect import Flow, apply_map, task


@task()
def do_something_one(item):
    return item + 1


@task()
def do_something_two(item, item_one):
    print(item, item_one)
    return item + item_one


@task()
def do_something_three(item_combination):
    print(item_combination)
    return item_combination[0] + item_combination[1]


# mapping function one
@task()
def do_something_with_one_two(item):
    result_one = do_something_one.run(item)
    result_two = do_something_two.run(item, result_one)
    return (result_one, result_two)


# mapping function two
@task()
def do_something_with_three(item):
    return do_something_three.run(item)


@task()
def print_results(r):
    print(r)
    return None


with Flow("test flow") as flow:
    l = [1]
    results = apply_map(do_something_with_one_two, l)
    results_results = apply_map(do_something_with_three, results)
    print_results(results_results)

flow.run()
```


By adding .run in the mapped tasks it works (and actually decorating them as a task)

**kevin701**: Hey <@U01EBUU7Y8N>, been looking at this. I’ll talk to the team about why this is cyclic.

**fabrice**: Quite curious about why that is happening! Keep me posted :slightly_smiling_face:

**kevin701**: <@ULVA73B9P> open “Bug with apply_map Causing Cyclical DAGs”

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1628683030208800?thread_ts=1628683030.208800&cid=CL09KU1K7).




Discussion:
I faced a similar issue while trying to send a python dictionary as an unmapped attribute to the task


```
from prefect import task, Flow, apply_map, unmapped
@task
def transform(result, credentials):
  print(credentials)
  result +=1
  return result 

with Flow("example") as flow:
  result = [1,2,3,4]
  credentials = {'id': ... , 'pass': ...}
  apply_map(transform ,result ,unmapped(credentials)) # can we do this ?
```


This raises :

> ValueError: Cycle found; flows must be acyclic!
Hi @abhasp 

`apply_map` is intended to be a non-task callable which _adds_ tasks to your flow

If you do `transform.map(result, unmapped(credentials))` you will get what you are looking for.

See this example from the `apply_map` docs

```python
from prefect import task, case, apply_map
from prefect.tasks.control_flow import merge

@task
def inc(x):
    return x + 1

@task
def is_even(x):
    return x % 2 == 0

def inc_if_even(x):
    with case(is_even(x), True):
        x2 = inc(x)
    return merge(x, x2)

with Flow("example") as flow:
    apply_map(inc_if_even, range(10))
```

And here is an example using `apply_map` to accomplish the same thing as `transform.map`

```python
from prefect import task, Flow, apply_map, unmapped


@task
def transform(result, credentials):
    print(credentials)
    result += 1
    return result


def map_transform(x, credentials):
    transform(x, credentials)


with Flow("example") as flow:
    result = [1, 2, 3, 4]
    credentials = {"id": ..., "pass": ...}
    apply_map(map_transform, result, unmapped(credentials))


Resolving patch:
diff --git a/src/prefect/utilities/tasks.py b/src/prefect/utilities/tasks.py
--- a/src/prefect/utilities/tasks.py
+++ b/src/prefect/utilities/tasks.py
@@ -77,6 +77,9 @@ def inc_if_even(x):
             raise ValueError("Couldn't infer a flow in the current context")
     assert isinstance(flow, prefect.Flow)  # appease mypy
 
+    # Cache the original tasks of the flow for calculating the difference later
+    original_flow_tasks = flow.tasks.copy()
+
     # Check if args/kwargs are valid first
     for x in itertools.chain(args, kwargs.values()):
         if not isinstance(
@@ -174,7 +177,7 @@ def preprocess(a: Any) -> "prefect.Task":
     #
     # Here we do a final pass adding missing upstream deps on mapped arguments
     # to all newly created tasks in the apply_map.
-    new_tasks = flow2.tasks.difference(arg_info)
+    new_tasks = flow2.tasks.difference(original_flow_tasks).difference(arg_info)
     for task in new_tasks:
         upstream_tasks = flow.upstream_tasks(task)
         is_root_in_subgraph = not upstream_tasks.intersection(new_tasks)
@@ -186,6 +189,7 @@ def preprocess(a: Any) -> "prefect.Task":
                     flow.add_edge(
                         upstream_task=arg_task, downstream_task=task, mapped=is_mapped
                     )
+
     return res
 
 


Test patch:
diff --git a/tests/utilities/test_tasks.py b/tests/utilities/test_tasks.py
--- a/tests/utilities/test_tasks.py
+++ b/tests/utilities/test_tasks.py
@@ -1,7 +1,7 @@
 import functools
 import pytest
 
-from prefect import Flow, Task, case, Parameter, resource_manager
+from prefect import Flow, Task, case, Parameter, resource_manager, task
 from prefect.engine.flow_runner import FlowRunner
 from prefect.engine.state import Paused, Resume
 from prefect.utilities import tasks, edges
@@ -358,6 +358,35 @@ def edge_info(task):
         }
         assert res == sol
 
+    def test_apply_map_can_be_used_on_apply_map_result(
+        self,
+    ):
+        # Prior to commit 4b0df740de99a1fad3182c01f4b182ba83445bcc this would introduce
+        # a cycle
+        @task
+        def combine(item_combination):
+            return item_combination[0] + item_combination[1]
+
+        # mapping function one
+        def apply_map_one(item):
+            result_one = inc(item)
+            result_two = inc(inc(item))
+            return (result_one, result_two)
+
+        # mapping function two
+        def apply_map_two(item):
+            return combine(item)
+
+        with Flow("test") as flow:
+            one_result = apply_map(apply_map_one, [1, 2, 3])
+            two_result = apply_map(apply_map_two, one_result)
+
+        state = flow.run()
+        assert state.is_successful()
+        assert state.result[one_result[0]].result == [2, 3, 4]
+        assert state.result[one_result[1]].result == [3, 4, 5]
+        assert state.result[two_result].result == [5, 7, 9]
+
     def test_apply_map_inside_case_statement_works(self):
         def func(x, a):
             return add(x, 1), add(x, a)