Repository: PrefectHQ/prefect

Issue:
Restart after modifying task state re-runs the failed task
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I refreshed the page and this issue still occurred.
- [X] I checked if this issue was specific to the browser I was using by testing with a different browser.

### Bug summary

Our team designs prefect flows in which tasks sometimes fail. When this happens, we perform some action (unrelated to prefect) and then set the state of the task to "completed", and continue running the flow. Our expectation is that the flow would continue from whichever task is after the failed task. However, we have noticed in Prefect 2 that retrying the flow after editing the failed tasks status to completed re-runs the failed task, and does not start from the next one. 

When we were using Prefect 1, I believe the behavior was what we expected, meaning that restarting a failed flow after manually editing the task's status in the UI would start the flow from the next task, and not re-run the failed task. So I am not sure if the different behavior we are experiencing with prefect 2 is a bug, or if there are other steps we need to take to replicate the behavior we expect. 



### Reproduction


1. Create a flow with three tasks, make the second task fail by always throwing an error or something. The flow should have persist_results=True. We also have remote_storage configured. 
```
from prefect import task, flow
from prefect.deployments.deployments import Deployment
from prefect.task_runners import SequentialTaskRunner
import asyncio

@task(persist_result=True)
def task_1():
    print("success!")

@task(persist_result=True)
def task_2():
    raise RuntimeError

@task(persist_result=True)
def task_3():
    print("success!")

@flow(task_runner=SequentialTaskRunner(), persist_result=True)
def demo_restart_issue():
    task_1()
    task_2()
    task_3()

async def deploy():
    local_deployment = await Deployment.build_from_flow(
        flow=demo_restart_issue,
        name=f"deployment_demo_restart_issue",
    )
    await local_deployment.apply()


if __name__ == "__main__": 
    asyncio.run(deploy())
```


3. Run the flow on the prefect UI. The flow will fail due to the second task failing.
4. Go to the second task in the UI, and manually set its status to "Completed". 
<img width="353" alt="Screen Shot 2023-09-18 at 9 09 31 AM" src="https://github.com/PrefectHQ/prefect/assets/29695959/4f99568e-c7ce-487c-827d-259b9c872aaf">

5. Restart the flow by clicking "Retry"
6. The flow will fail again because it will re-run the second task, even though we set its state to completed. 


### Error

Prefect UI displaying initial task and flow run failure: 
<img width="1070" alt="image" src="https://github.com/PrefectHQ/prefect/assets/29695959/e063829b-d9f1-4291-b0a3-082de3eae828">

This is how I modified the state of task run 2:
<img width="1075" alt="Screen Shot 2023-09-18 at 9 43 29 AM" src="https://github.com/PrefectHQ/prefect/assets/29695959/e8a362f2-fd32-4381-a8bf-d7a4e790e2ab">


The flow run after modifying the state of the task, but before retrying the flow:
<img width="1066" alt="Screen Shot 2023-09-18 at 9 44 01 AM" src="https://github.com/PrefectHQ/prefect/assets/29695959/298ebaa6-0bd6-4a0c-b79e-d577e03f71d6">


Prefect UI after clicking "Retry". I would have instead inspected the flow to restart from **after** task 2, and run task 3. Instead, it retries task 2, which fails again.
<img width="1069" alt="Screen Shot 2023-09-18 at 9 46 17 AM" src="https://github.com/PrefectHQ/prefect/assets/29695959/001b36e3-aa4d-4c12-a5d4-e6182e011d56">





### Browers

- [X] Chrome
- [ ] Firefox
- [X] Safari
- [ ] Edge

### Prefect version

```Text
Version:             2.13.0
API version:         0.8.4
Python version:      3.10.12
Git commit:          66577cc6
Built:               Thu, Sep 7, 2023 3:52 PM
OS/Arch:             darwin/arm64
Profile:             ephemeral
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.42.0
```
```


### Additional context

_No response_


Discussion:
+1 . I face the same issue. hope this can be fixed since its impossible to skip tasks in that case.
Hey thanks for filing this issue and the great writeup! This behavior is not strictly a bug, however it is maybe a bit unexpected coming from Prefect1. We'll need to do some investigation into supporting this
> Hey thanks for filing this issue and the great writeup! This behavior is not strictly a bug, however it is maybe a bit unexpected coming from Prefect1. We'll need to do some investigation into supporting this

Thanks! I'm confused about why this behavior is not classified as a bug though. When we retry a flow run, any tasks that already completed successfully are skipped, **except** for tasks that were marked completed manually. This distinction in behavior means that the ability to mark tasks as completed manually is essentially pointless. Unless the manual completed setting is intended to be used in a different way than I understand it?
+1. Our use-case: sometimes a task fails and we might need to manually perform the task but still want the rest of the flow steps to continue _after_ the manual intervention
Small update: I've looked into this and agree this is a use case we need to support.  It appears that we over-indexed to situations where data is exchanged between tasks, so anytime a completed state has an empty data payload Prefect decides to rerun the task.  I tried doing a few simple adjustments, but those caused other downstream errors so I'll need to be careful about modifying the result logic.

I think the ideal behavior would be to not rerun completed tasks and to only raise an error if a downstream task actually consumes the output of an upstream task with no persisted result payload.
Thanks for the update @cicdw! 

> It appears that we over-indexed to situations where data is exchanged between tasks, so anytime a completed state has an empty data payload Prefect decides to rerun the task.

Does this mean that this behavior only happens when a task has an "empty data payload"? I'm not sure exactly what that means -- does it have to do with the task's `Result`? If that's the case, then can we ensure completed tasks get properly skipped during flow run restarts by ensuring that all tasks return a value that is consumed by a downstream task?



Resolving patch:
diff --git a/src/prefect/engine.py b/src/prefect/engine.py
--- a/src/prefect/engine.py
+++ b/src/prefect/engine.py
@@ -165,7 +165,7 @@
     patch_print,
     task_run_logger,
 )
-from prefect.results import BaseResult, ResultFactory
+from prefect.results import BaseResult, ResultFactory, UnknownResult
 from prefect.settings import (
     PREFECT_DEBUG_MODE,
     PREFECT_LOGGING_LOG_PRINTS,
@@ -174,6 +174,7 @@
     PREFECT_UI_URL,
 )
 from prefect.states import (
+    Completed,
     Paused,
     Pending,
     Running,
@@ -1935,6 +1936,22 @@ async def orchestrate_task_run(
     )
     last_state = task_run.state
 
+    # Completed states with persisted results should have result data. If it's missing,
+    # this could be a manual state transition, so we should use the Unknown result type
+    # to represent that we know we don't know the result.
+    if (
+        last_state
+        and last_state.is_completed()
+        and result_factory.persist_result
+        and not last_state.data
+    ):
+        state = await propose_state(
+            client,
+            state=Completed(data=await UnknownResult.create()),
+            task_run_id=task_run.id,
+            force=True,
+        )
+
     # Transition from `PENDING` -> `RUNNING`
     try:
         state = await propose_state(
diff --git a/src/prefect/results.py b/src/prefect/results.py
--- a/src/prefect/results.py
+++ b/src/prefect/results.py
@@ -613,3 +613,41 @@ class PersistedResultBlob(pydantic.BaseModel):
 
     def to_bytes(self) -> bytes:
         return self.json().encode()
+
+
+class UnknownResult(BaseResult):
+    """
+    Result type for unknown results. Typipcally used to represent the result
+    of tasks that were forced from a failure state into a completed state.
+
+    The value for this result is always None and is not persisted to external
+    result storage, but orchestration treats the result the same as persisted
+    results when determining orchestration rules, such as whether to rerun a
+    completed task.
+    """
+
+    type = "unknown"
+    value: None
+
+    def has_cached_object(self) -> bool:
+        # This result type always has the object cached in memory
+        return True
+
+    @sync_compatible
+    async def get(self) -> R:
+        return self.value
+
+    @classmethod
+    @sync_compatible
+    async def create(
+        cls: "Type[UnknownResult]",
+        obj: R = None,
+    ) -> "UnknownResult[R]":
+        if obj is not None:
+            raise TypeError(
+                f"Unsupported type {type(obj).__name__!r} for unknown result. "
+                "Only None is supported."
+            )
+
+        description = "Unknown result persisted to Prefect."
+        return cls(value=obj, artifact_type="result", artifact_description=description)
diff --git a/src/prefect/server/orchestration/core_policy.py b/src/prefect/server/orchestration/core_policy.py
--- a/src/prefect/server/orchestration/core_policy.py
+++ b/src/prefect/server/orchestration/core_policy.py
@@ -13,6 +13,7 @@
 from packaging.version import Version
 from sqlalchemy import select
 
+from prefect.results import UnknownResult
 from prefect.server import models
 from prefect.server.database.dependencies import inject_db
 from prefect.server.database.interface import PrefectDBInterface
@@ -76,13 +77,16 @@ def priority():
 
 class MinimalFlowPolicy(BaseOrchestrationPolicy):
     def priority():
-        return []
+        return [
+            AddUnknownResult,  # mark forced completions with an unknown result
+        ]
 
 
 class MinimalTaskPolicy(BaseOrchestrationPolicy):
     def priority():
         return [
             ReleaseTaskConcurrencySlots,  # always release concurrency slots
+            AddUnknownResult,  # mark forced completions with a result placeholder
         ]
 
 
@@ -194,6 +198,45 @@ async def after_transition(
                 cl.active_slots = list(active_slots)
 
 
+class AddUnknownResult(BaseOrchestrationRule):
+    """
+    Assign an "unknown" result to runs that are forced to complete from a
+    failed or crashed state, if the previous state used a persisted result.
+
+    When we retry a flow run, we retry any task runs that were in a failed or
+    crashed state, but we also retry completed task runs that didn't use a
+    persisted result. This means that without a sentinel value for unknown
+    results, a task run forced into Completed state will always get rerun if the
+    flow run retries because the task run lacks a persisted result. The
+    "unknown" sentinel ensures that when we see a completed task run with an
+    unknown result, we know that it was forced to complete and we shouldn't
+    rerun it.
+
+    Flow runs forced into a Completed state have a similar problem: without a
+    sentinel value, attempting to refer to the flow run's result will raise an
+    exception because the flow run has no result. The sentinel ensures that we
+    can distinguish between a flow run that has no result and a flow run that
+    has an unknown result.
+    """
+
+    FROM_STATES = [StateType.FAILED, StateType.CRASHED]
+    TO_STATES = [StateType.COMPLETED]
+
+    async def before_transition(
+        self,
+        initial_state: Optional[states.State],
+        proposed_state: Optional[states.State],
+        context: TaskOrchestrationContext,
+    ) -> None:
+        if (
+            initial_state
+            and initial_state.data
+            and initial_state.data.get("type") == "reference"
+        ):
+            unknown_result = await UnknownResult.create()
+            self.context.proposed_state.data = unknown_result.dict()
+
+
 class CacheInsertion(BaseOrchestrationRule):
     """
     Caches completed states with cache keys after they are validated.


Test patch:
diff --git a/tests/results/test_unknown_results.py b/tests/results/test_unknown_results.py
new file mode 100644
--- /dev/null
+++ b/tests/results/test_unknown_results.py
@@ -0,0 +1,62 @@
+import json
+
+import pytest
+
+from prefect.results import BaseResult, UnknownResult
+
+INVALID_VALUES = [True, False, "hey"]
+
+
+@pytest.mark.parametrize("value", INVALID_VALUES)
+async def test_unknown_result_invalid_values(value):
+    with pytest.raises(TypeError, match="Unsupported type"):
+        await UnknownResult.create(value)
+
+
+def test_unknown_result_create_and_get_sync():
+    result = UnknownResult.create()
+    assert result.get() is None
+
+
+async def test_unknown_result_create_and_get_async():
+    result = await UnknownResult.create()
+    assert await result.get() is None
+
+
+def test_unknown_result_create_and_get_with_explicit_value():
+    result = UnknownResult.create(obj=None)
+    assert result.get() is None
+
+
+async def test_result_unknown_json_roundtrip():
+    result = await UnknownResult.create()
+    serialized = result.json()
+    deserialized = UnknownResult.parse_raw(serialized)
+    assert await deserialized.get() is None
+
+
+async def test_unknown_result_json_roundtrip_base_result_parser():
+    result = await UnknownResult.create()
+    serialized = result.json()
+    deserialized = BaseResult.parse_raw(serialized)
+    assert await deserialized.get() is None
+
+
+async def test_unknown_result_populates_default_artifact_metadata():
+    result = await UnknownResult.create()
+    assert result.artifact_type == "result"
+    assert result.artifact_description == "Unknown result persisted to Prefect."
+
+
+async def test_unknown_result_null_is_distinguishable_from_none():
+    """
+    This is important for separating cases where _no result_ is stored in the database
+    because the user disabled persistence (for example) from cases where the result
+    is stored but is a null value.
+    """
+    result = await UnknownResult.create(None)
+    assert result is not None
+    serialized = result.json()
+    assert serialized is not None
+    assert serialized != "null"
+    assert json.loads(serialized) is not None
diff --git a/tests/server/orchestration/test_core_policy.py b/tests/server/orchestration/test_core_policy.py
--- a/tests/server/orchestration/test_core_policy.py
+++ b/tests/server/orchestration/test_core_policy.py
@@ -8,11 +8,17 @@
 import pendulum
 import pytest
 
-from prefect.results import LiteralResult, PersistedResult, UnpersistedResult
+from prefect.results import (
+    LiteralResult,
+    PersistedResult,
+    UnknownResult,
+    UnpersistedResult,
+)
 from prefect.server import schemas
 from prefect.server.exceptions import ObjectNotFoundError
 from prefect.server.models import concurrency_limits, flow_runs
 from prefect.server.orchestration.core_policy import (
+    AddUnknownResult,
     BypassCancellingScheduledFlowRuns,
     CacheInsertion,
     CacheRetrieval,
@@ -1248,7 +1254,9 @@ async def test_transitions_from_completed_to_non_final_states_allowed_without_pe
         ],
         ids=transition_names,
     )
-    @pytest.mark.parametrize("result_type", [PersistedResult, LiteralResult])
+    @pytest.mark.parametrize(
+        "result_type", [PersistedResult, LiteralResult, UnknownResult]
+    )
     async def test_transitions_from_completed_to_non_final_states_rejected_with_persisted_result(
         self,
         session,
@@ -2957,3 +2965,68 @@ async def test_accepts_cancelling_flow_run_with_pid(
 
         assert ctx.response_status == SetStateStatus.ACCEPT
         assert ctx.validated_state_type == states.StateType.CANCELLING
+
+
+@pytest.mark.parametrize("run_type", ["task", "flow"])
+class TestAddUnknownResultRule:
+    @pytest.mark.parametrize(
+        "result_type,initial_state_type",
+        list(
+            product(
+                (PersistedResult,), (states.StateType.FAILED, states.StateType.CRASHED)
+            )
+        ),
+    )
+    async def test_saves_unknown_result_if_last_state_used_persisted_results(
+        self,
+        session,
+        initialize_orchestration,
+        run_type,
+        result_type,
+        initial_state_type,
+    ):
+        proposed_state_type = states.StateType.COMPLETED
+        intended_transition = (initial_state_type, proposed_state_type)
+        ctx = await initialize_orchestration(
+            session,
+            run_type,
+            *intended_transition,
+            initial_state_data=result_type.construct().dict() if result_type else None,
+        )
+
+        async with AddUnknownResult(ctx, *intended_transition) as ctx:
+            await ctx.validate_proposed_state()
+
+        assert ctx.proposed_state.data.get("type") == "unknown"
+
+    @pytest.mark.parametrize(
+        "result_type,initial_state_type",
+        list(
+            product(
+                (UnpersistedResult, LiteralResult),
+                (states.StateType.FAILED, states.StateType.CRASHED),
+            )
+        ),
+    )
+    async def test_does_not_save_unknown_result_if_last_result_did_not_use_persisted_results(
+        self,
+        session,
+        initialize_orchestration,
+        run_type,
+        result_type,
+        initial_state_type,
+    ):
+        proposed_state_type = states.StateType.COMPLETED
+        intended_transition = (initial_state_type, proposed_state_type)
+        ctx = await initialize_orchestration(
+            session,
+            run_type,
+            *intended_transition,
+            initial_state_data=result_type.construct().dict() if result_type else None,
+        )
+
+        async with AddUnknownResult(ctx, *intended_transition) as ctx:
+            await ctx.validate_proposed_state()
+
+        if ctx.proposed_state.data:
+            assert ctx.proposed_state.data.get("type") != "unknown"
diff --git a/tests/test_engine.py b/tests/test_engine.py
--- a/tests/test_engine.py
+++ b/tests/test_engine.py
@@ -23,7 +23,7 @@
 import prefect.flows
 from prefect import engine, flow, task
 from prefect._internal.compatibility.experimental import ExperimentalFeature
-from prefect.client.orchestration import get_client
+from prefect.client.orchestration import PrefectClient, get_client
 from prefect.client.schemas import OrchestrationResult
 from prefect.context import FlowRunContext, get_run_context
 from prefect.engine import (
@@ -54,7 +54,7 @@
 )
 from prefect.futures import PrefectFuture
 from prefect.input import RunInput, read_flow_run_input
-from prefect.results import ResultFactory
+from prefect.results import ResultFactory, UnknownResult
 from prefect.server.schemas.core import FlowRun
 from prefect.server.schemas.filters import FlowRunFilter
 from prefect.server.schemas.responses import (
@@ -1984,6 +1984,49 @@ def my_task(x):
             in caplog.text
         )
 
+    async def test_proposes_unknown_result_if_state_is_completed_and_result_data_is_missing(
+        self,
+        mock_anyio_sleep,
+        prefect_client: PrefectClient,
+        flow_run,
+        result_factory,
+        local_filesystem,
+    ):
+        # the flow run must be running prior to running tasks
+        await prefect_client.set_flow_run_state(
+            flow_run_id=flow_run.id,
+            state=Running(),
+        )
+
+        @task()
+        def my_task():
+            return 1
+
+        # Create a task run to test
+        task_run = await prefect_client.create_task_run(
+            task=my_task,
+            flow_run_id=flow_run.id,
+            state=Completed(data=None),
+            dynamic_key="0",
+        )
+
+        result_factory.persist_result = True
+
+        state = await orchestrate_task_run(
+            task=my_task,
+            task_run=task_run,
+            parameters={},
+            wait_for=None,
+            result_factory=result_factory,
+            interruptible=False,
+            client=prefect_client,
+            log_prints=False,
+        )
+
+        result = await state.result()
+        assert result is None
+        assert isinstance(state.data, UnknownResult)
+
 
 class TestBeginTaskRun:
     async def test_begin_task_run_handles_pause_signal(