Repository: PrefectHQ/prefect

Issue:
Proposal: make upstream inputs have precedance over cached_inputs
Currently, the `TaskRunner.get_task_inputs()` method will load upstream inputs, then overwrite that dict with any cached inputs on the task. I'd like to propose reversing that order, so first cached inputs are loaded, then upstream inputs overwrite them.

Let's say task A runs and then task B wants to be retried. If I (manually) rerun task A, then my expectation would be that task B would use A's *new* result, not the old one that task B cached. In other words, while it's a useful and important Prefect feature that *B* does the caching of *A's* result, this is a case where that would go against my expectation, since B's cache no longer matches A's result.

This isn't a fully baked recommendation, but I'd like to think through the above situation and see if it can/should be addressed. 

cc @cicdw 


Discussion:
> If I (manually) rerun task A, then my expectation would be that task B would use A's new result, not the old one that task B cached.

What if you don't request caching on A?  Where would this new result be retrieved from?  And how would you tell the difference between a `None` result on an upstream caused by no-caching vs. a `None` result which is intentional?
Those are good questions. In this scenario, the following steps have taken place:

1. A flow run started and A ran, then B ran and wants to be retried.
2. I now put A back in a `Pending` state and run the flow again
3. The flow will evaluate A and run it to completion, producing a result. It will then move to B and run B with A's result in the `upstream_states`, but it will ignore the result that A just produced in this very same run and prefer the cached result from B's *previous* run. I think that behavior is incorrect: A just passed a result directly to B, and B should use it. 

`None` is an excellent question, though. I'm not sure I have an answer immediately. 🤔 
Ok, so I was intrigued by this and so I made the following change:
```diff
diff --git a/src/prefect/engine/task_runner.py b/src/prefect/engine/task_runner.py
index 36f63090..13f6e5fd 100644
--- a/src/prefect/engine/task_runner.py
+++ b/src/prefect/engine/task_runner.py
@@ -598,11 +598,12 @@ class TaskRunner(Runner):
 
         for edge, upstream_state in upstream_states.items():
             # construct task inputs
-            if edge.key is not None:
+            if edge.key is not None and upstream_state.result is not None:
                 task_inputs[edge.key] = upstream_state.result
 
         if state.is_pending() and state.cached_inputs is not None:  # type: ignore
-            task_inputs.update(state.cached_inputs)  # type: ignore
+            for key, val in state.cached_inputs.items():
+                task_inputs.setdefault(key, val)
 
         return task_inputs
```
and ran all tests; only _two_ failed:
- `test_merge_diamond_flow_with_results`
- `test_get_inputs_from_cached_inputs_overwrites_upstream_states`

No prizes for guessing the second one would fail!

So I think this is do-able, but the issue with `None` is super tricky.  To articulate for posterity, upstream state results will be `None` if the state was never populated with data (as would be the case if a user was running the second run of a flow and the only task that needed running was a `Retrying` task).  This makes it difficult to determine whether an upstream result is `None` because the task didn't run in the current run, or is `None` because the task _did_ run in the current session and its result was ... well, `None`.

💡 💡 💡 💡 💡 💡 💡 💡 💡 💡time!  If we implement #596 and upstream states have fully hydrated `Result()` objects attached to them, we could implement a special class of `Result` called `NoResult` or something to that effect.  This would allow us to distinguish between `None` as a completely reasonable result of a task vs. a true _absence_ of result.

Resolving patch:
diff --git a/src/prefect/engine/task_runner.py b/src/prefect/engine/task_runner.py
--- a/src/prefect/engine/task_runner.py
+++ b/src/prefect/engine/task_runner.py
@@ -538,8 +538,8 @@ def get_task_inputs(
     ) -> Dict[str, Result]:
         """
         Given the task's current state and upstream states, generates the inputs for this task.
-        If the current state has `cached_inputs`, they are used. Upstream states supplement
-        any missing keys.
+        Upstream state result values are used. If the current state has `cached_inputs`, they
+        will override any upstream values which are `NoResult`.
 
         Args:
             - state (State): the task's current state.
@@ -557,9 +557,13 @@ def get_task_inputs(
                 task_inputs[edge.key] = upstream_state._result.read()
 
         if state.is_pending() and state.cached_inputs is not None:  # type: ignore
-            task_inputs.update(  # type: ignore
-                {k: r.read() for k, r in state.cached_inputs.items()}  # type: ignore
-            )  # type: ignore
+            task_inputs.update(
+                {
+                    k: r.read()
+                    for k, r in state.cached_inputs.items()  # type: ignore
+                    if task_inputs.get(k, NoResult) == NoResult
+                }
+            )
 
         return task_inputs
 


Test patch:
diff --git a/tests/engine/cloud/test_cloud_task_runner.py b/tests/engine/cloud/test_cloud_task_runner.py
--- a/tests/engine/cloud/test_cloud_task_runner.py
+++ b/tests/engine/cloud/test_cloud_task_runner.py
@@ -345,8 +345,8 @@ def add(x, y):
         x = Result(1, handled=False, result_handler=JSONResultHandler())
         y = Result("0", handled=False, result_handler=JSONResultHandler())
         state = Pending(cached_inputs=dict(x=x, y=y))
-        x_state = Success(result=1)
-        y_state = Success(result=1)
+        x_state = Success()
+        y_state = Success()
         upstream_states = {
             Edge(Task(), Task(), key="x"): x_state,
             Edge(Task(), Task(), key="y"): y_state,
diff --git a/tests/engine/test_task_runner.py b/tests/engine/test_task_runner.py
--- a/tests/engine/test_task_runner.py
+++ b/tests/engine/test_task_runner.py
@@ -675,16 +675,26 @@ def test_get_inputs_from_cached_inputs_and_upstream_states(self):
         )
         assert inputs == {"x": Result(1), "y": Result(2)}
 
-    def test_get_inputs_from_cached_inputs_overwrites_upstream_states(self):
+    def test_cached_inputs_overwrites_upstream_states_if_they_are_noresult(self):
         inputs = TaskRunner(task=Task()).get_task_inputs(
-            state=Pending(cached_inputs={"x": Result(1)}),
+            state=Pending(cached_inputs={"x": Result(1), "y": Result(2)}),
             upstream_states={
-                Edge(1, 2, key="x"): Success(result=2),
-                Edge(1, 2, key="y"): Success(result=2),
+                Edge(1, 2, key="x"): Success(),
+                Edge(1, 2, key="y"): Success(),
             },
         )
         assert inputs == {"x": Result(1), "y": Result(2)}
 
+    def test_cached_inputs_doesnt_overwrites_upstream_states_if_hydrated(self):
+        inputs = TaskRunner(task=Task()).get_task_inputs(
+            state=Pending(cached_inputs={"x": Result(1), "y": Result(2)}),
+            upstream_states={
+                Edge(1, 2, key="x"): Success(result=None),
+                Edge(1, 2, key="y"): Success(result=5),
+            },
+        )
+        assert inputs == {"x": Result(None), "y": Result(5)}
+
 
 class TestCheckTaskCached:
     @pytest.mark.parametrize("state", [Pending(), Success(), Retrying()])