Repository: PrefectHQ/prefect

Issue:
FlowRunner manages context cache wrongly
## Description
AttributeError and different behavior of output caching between Flow.run and FlowRunner.run.

## Expected Behavior
No AttributeError, same behavior and cached result reuse of FlowRunner.run after Flow.run.

## Reproduction
```python
from datetime import timedelta
import random

from prefect import task, Flow
from prefect.engine import FlowRunner
from prefect.engine.cache_validators import duration_only

@task(cache_for=timedelta(seconds=10), 
      cache_validator=duration_only)
def rand_inc(r, x):
    rand = random.randint(0, r)
    print("RAND", rand)
    return rand + x

with Flow("Cache") as f:
    a1 = rand_inc(10, 0)

# this fails:
runner = FlowRunner(f)
state1 = runner.run()

# this would pass:
#f.run()
```
Traceback:
```
Traceback (most recent call last):
  File "/Users/dafcok/miniconda3/lib/python3.6/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "~/miniconda3/lib/python3.6/site-packages/prefect/engine/task_runner.py", line 631, in check_task_is_cached
    candidate_states = prefect.context.caches.get(
AttributeError: 'Context' object has no attribute 'caches'
```

Moreover, if you `f.run()` once before `runner.run()`, cache validation always logs `Task 'rand_inc': can't use cache because it is now invalid`.

## Environment
```
{
  "config_overrides": {},
  "env_vars": [],
  "system_information": {
    "platform": "Darwin-19.0.0-x86_64-i386-64bit",
    "prefect_version": "0.10.4",
    "python_version": "3.6.8"
  }
}
```


Discussion:
Looking for some thoughts on this. What's happening here is the `flow.run` function sets a global `caches` block in context (among other things)

https://github.com/PrefectHQ/prefect/blob/6d141372cf89064d24bbafad582a9db26be0cbd5/src/prefect/core/flow.py#L1024-L1025

And then is uses flow runners to run that flow where each flow runner would have access to that `context.caches`. Now running the flow directly from the flow runner takes it a level deeper, below that context block. This means that if the flow runner is updated to make this `context.caches` in a similar way to the `flow.run` then the cache will not persist across runs since the flow runner is responsible for only a single run. We could add this in there with the knowledge that it will proceed without error but in calling something like 

```
runner = FlowRunner(f)
state1 = runner.run()
state2 = runner.run()
```

the second run will not use the cache from the first run.
I think one design goal could be that FlowRunner.run() is guaranteed to not modify global state, caches or otherwise. Is that already the case? If yes, passing `context=my_dict` could explicitly modify `my_dict` which can be used for other runs.
In the long run, I'm not sure if in-memory caching should become obsolete after #2394 

Resolving patch:
diff --git a/src/prefect/engine/task_runner.py b/src/prefect/engine/task_runner.py
--- a/src/prefect/engine/task_runner.py
+++ b/src/prefect/engine/task_runner.py
@@ -637,9 +637,11 @@ def check_task_is_cached(self, state: State, inputs: Dict[str, Result]) -> State
                 state = Pending("Cache was invalid; ready to run.")
 
         if self.task.cache_for is not None:
-            candidate_states = prefect.context.caches.get(
-                self.task.cache_key or self.task.name, []
-            )
+            candidate_states = []
+            if prefect.context.get("caches"):
+                candidate_states = prefect.context.caches.get(
+                    self.task.cache_key or self.task.name, []
+                )
             sanitized_inputs = {key: res.value for key, res in inputs.items()}
             for candidate in candidate_states:
                 if self.task.cache_validator(


Test patch:
diff --git a/tests/engine/test_flow_runner.py b/tests/engine/test_flow_runner.py
--- a/tests/engine/test_flow_runner.py
+++ b/tests/engine/test_flow_runner.py
@@ -690,6 +690,24 @@ def run(self, x, s):
         assert flow_state.result[y].result == 100
 
 
+class TestCachingFromContext:
+    def test_caches_do_not_persist_across_flow_runner_runs(self):
+        @prefect.task(cache_for=datetime.timedelta(seconds=10))
+        def test_task():
+            return random.random()
+
+        with Flow("test_cache") as flow:
+            t = test_task()
+
+        flow_state = FlowRunner(flow=flow).run(return_tasks=[t])
+        first_result = flow_state.result[t].result
+
+        flow_state = FlowRunner(flow=flow).run(return_tasks=[t])
+        second_result = flow_state.result[t].result
+
+        assert first_result != second_result
+
+
 class TestInitializeRun:
     def test_initialize_sets_none_to_pending(self):
         result = FlowRunner(Flow(name="test")).initialize_run(