Repository: PrefectHQ/prefect

Issue:
DaskExecutor doesn't write results
## Description
DaskExecutor doesn't write LocalResults with JSONSerializer.

In our tests, also other results and with a cluster aren't written. For the Repro, I picked the simplest case.
## Expected Behavior
DaskExecutor writes LocalResults. If workers need extra treatment to set their global checkpointing config, I'd consider it an enhancement if configs percolate.

## Reproduction
```python
import json
import os
from tempfile import TemporaryDirectory
from typing import Optional

from prefect import Flow, task
from prefect.engine.executors import Executor, DaskExecutor
from prefect.engine.results import LocalResult
from prefect.engine.serializers import JSONSerializer
from prefect.utilities.configuration import set_temporary_config
from prefect.utilities.debug import raise_on_exception

def test(e: Optional[Executor]):
    with TemporaryDirectory() as tmpdir:
        flow_result = LocalResult(tmpdir, serializer=JSONSerializer(),
                                  location="{task_name}.json")

        with Flow("write_result", result=flow_result) as f:
            _terminal = task(lambda: 42, checkpoint=True, name="magic")()

        with set_temporary_config({"flows.checkpointing": True}), \
             raise_on_exception():
            f.run(executor=e)

        files = os.listdir(tmpdir)
        assert files == ["magic.json"], files
        with open(os.path.join(tmpdir, files[0]), "rb") as file:
            val = json.load(file)
        assert val==42

if __name__ == "__main__":
    print("Local")
    test(None)
    print("DaskExecutor")
    test(DaskExecutor())
```
*Output:*
```bash
Local
[2020-08-01 09:48:05] INFO - prefect.FlowRunner | Beginning Flow run for 'write_result'
[2020-08-01 09:48:05] INFO - prefect.FlowRunner | Starting flow run.
[2020-08-01 09:48:05] INFO - prefect.TaskRunner | Task 'magic': Starting task run...
[2020-08-01 09:48:05] INFO - prefect.TaskRunner | Task 'magic': finished task run for task with final state: 'Success'
[2020-08-01 09:48:05] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
DaskExecutor
[2020-08-01 09:48:05] INFO - prefect.FlowRunner | Beginning Flow run for 'write_result'
[2020-08-01 09:48:05] INFO - prefect.FlowRunner | Starting flow run.
[2020-08-01 09:48:07] INFO - prefect.TaskRunner | Task 'magic': Starting task run...
[2020-08-01 09:48:07] INFO - prefect.TaskRunner | Task 'magic': finished task run for task with final state: 'Success'
[2020-08-01 09:48:08] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
Traceback (most recent call last):
  File "repro_local_write.py", line 37, in <module>
    test(DaskExecutor())
  File "repro_local_write.py", line 26, in test
    assert files == ["magic.json"], files
AssertionError: []
```

## Environment
```json
{
  "config_overrides": {},
  "env_vars": [],
  "system_information": {
    "platform": "Darwin-19.5.0-x86_64-i386-64bit",
    "prefect_version": "0.12.5",
    "python_version": "3.6.8"
  }
}
```
`dask==2.21.0`


Discussion:
Hi @ahirner I don't think setting temporary config like that will do what you expect due to the workers being created outside of that context manager. Having checkpointing set in your global config.toml (or through an env var) should do the trick:

```
[flows]
checkpointing = true
```
Global configuration options work, thanks! I didn't expect clients to have only partial control over checkpointing.

Resolving patch:
diff --git a/src/prefect/engine/task_runner.py b/src/prefect/engine/task_runner.py
--- a/src/prefect/engine/task_runner.py
+++ b/src/prefect/engine/task_runner.py
@@ -159,7 +159,12 @@ def initialize_run(  # type: ignore
             task_name=self.task.name,
             task_tags=self.task.tags,
         )
-        context.setdefault("checkpointing", config.flows.checkpointing)
+        # Use the config stored in context if possible (should always be present)
+        try:
+            checkpointing = context["config"]["flows"]["checkpointing"]
+        except KeyError:
+            checkpointing = config.flows.checkpointing
+        context.setdefault("checkpointing", checkpointing)
 
         map_index = context.get("map_index", None)
         if isinstance(map_index, int) and context.get("task_full_name"):


Test patch:
diff --git a/tests/engine/test_task_runner.py b/tests/engine/test_task_runner.py
--- a/tests/engine/test_task_runner.py
+++ b/tests/engine/test_task_runner.py
@@ -435,6 +435,13 @@ def test_task_runner_puts_checkpointing_in_context(self):
         with prefect.context() as ctx:
             assert "checkpointing" not in ctx
             with set_temporary_config({"flows.checkpointing": "FOO"}):
+                # Pull from context.config if present
+                prefect.config.flows.checkpointing = False
+                result = TaskRunner(Task()).initialize_run(state=None, context=ctx)
+                assert result.context.checkpointing == "FOO"
+                # Otherwise fallback to local config
+                prefect.config.flows.checkpointing = "FOO"
+                del prefect.context.config.flows.checkpointing
                 result = TaskRunner(Task()).initialize_run(state=None, context=ctx)
                 assert result.context.checkpointing == "FOO"