Repository: PrefectHQ/prefect

Issue:
Task Run Fails to Read Upstream Result the First Time a Flow is Run on Windows
## Description
<!-- A clear description of the bug -->
![image](https://user-images.githubusercontent.com/2325367/94076056-4faa3f80-fdca-11ea-9269-56edfae373bf.png)
> it seems that the file is either getting cleaned up OR reading from the file has a bug that is specific to windows paths



## Expected Behavior
<!-- What did you expect to happen instead? -->
Result is read properly


## Reproduction
<!-- A minimal example that exhibits the behavior. -->




## Environment
<!-- Any additional information about your environment

Optionally run `prefect diagnostics` from the command line and paste the information here. -->
Windows


Discussion:
@dylanbhughes I am unable to reproduce on a Windows machine; I'll need some example code + configuration.  Note that this could happen if you are using caching (`cache_for`) across multiple machines with a `LocalResult`
OK after attempting to reproduce this in a few different ways, I've concluded that _something_ external to Prefect must have deleted this file.  Based on the traceback, it's clear this Task used caching, and the non-existence of the file meant that this particular `Cached` state could not be used.  

I _do_ think Prefect should have more gracefully handled this situation by logging a warning that we were unable to read the result and therefore will not be using that particular cache.  There's no reason to fail the execution of the task here, because nonexistence of the file is equivalent to the cache being invalid IMO.  Leaving this ticket open to track this UX enhancement.

Resolving patch:
diff --git a/src/prefect/engine/cloud/task_runner.py b/src/prefect/engine/cloud/task_runner.py
--- a/src/prefect/engine/cloud/task_runner.py
+++ b/src/prefect/engine/cloud/task_runner.py
@@ -266,7 +266,16 @@ def check_task_is_cached(self, state: State, inputs: Dict[str, Result]) -> State
                         sanitized_inputs,
                         prefect.context.get("parameters"),
                     ):
-                        return candidate_state.load_result(self.result)
+                        try:
+                            return candidate_state.load_result(self.result)
+                        except Exception:
+                            location = getattr(
+                                candidate_state._result, "location", None
+                            )
+                            self.logger.warning(
+                                f"Failed to load cached state data from {location}.",
+                                exc_info=True,
+                            )
 
                 self.logger.debug(
                     "Task '{name}': can't use cache because no candidate Cached states "


Test patch:
diff --git a/tests/engine/cloud/test_cloud_task_runner.py b/tests/engine/cloud/test_cloud_task_runner.py
--- a/tests/engine/cloud/test_cloud_task_runner.py
+++ b/tests/engine/cloud/test_cloud_task_runner.py
@@ -13,7 +13,7 @@
 from prefect.engine.cache_validators import all_inputs, duration_only
 from prefect.engine.cloud import CloudTaskRunner
 from prefect.engine.result import NoResult, Result, SafeResult, NoResult
-from prefect.engine.results import PrefectResult, SecretResult
+from prefect.engine.results import PrefectResult, SecretResult, LocalResult
 
 from prefect.engine.result_handlers import JSONResultHandler, ResultHandler
 from prefect.engine.runner import ENDRUN
@@ -253,6 +253,32 @@ def cached_task():
     assert res.result == 42
 
 
+def test_task_runner_treats_unfound_files_as_invalid_caches(client, tmpdir):
+    @prefect.task(
+        cache_for=datetime.timedelta(minutes=1), result_handler=JSONResultHandler()
+    )
+    def cached_task():
+        return 42
+
+    state = Cached(
+        cached_result_expiration=datetime.datetime.utcnow()
+        + datetime.timedelta(minutes=2),
+        result=LocalResult(location=str(tmpdir / "made_up_data.prefect")),
+    )
+    old_state = Cached(
+        cached_result_expiration=datetime.datetime.utcnow()
+        + datetime.timedelta(days=1),
+        result=Result(13, JSONResultHandler()),
+    )
+    client.get_latest_cached_states = MagicMock(return_value=[state, old_state])
+
+    res = CloudTaskRunner(task=cached_task).run()
+    assert client.get_latest_cached_states.called
+    assert res.is_successful()
+    assert res.is_cached()
+    assert res.result == 13
+
+
 class TestCheckTaskCached:
     def test_reads_result_if_cached_valid(self, client):
         result = PrefectResult(location="2")