Repository: PrefectHQ/prefect

Issue:
Allow for volume mounts on Docker storage and / or Docker agent
We should allow for volume mounts to flows running with Docker storage.  I believe this needs to be an option on the Docker _agent_ (not the storage class), as it won't make sense in other environments (e.g., K8s).

This will allow for users to use e.g., `LocalResultHandler` within Docker storage and still write to local disk.


Discussion:
I'll move forward under the assumption that the agent owns volume attachment, signaled by labels. Specifically, the agent would mount volumes supplied at start up to all Flows the agent acts on, it would be up to the user to add labels to the agent at startup or flow at registration to select which flows to run on the agent:

```bash
# option a
prefect agent start docker --volume /some/path/to/x:/ctr/path --label provides-x-volume
... and ...
flow.register(labels=['provides-x-volume'], ...)
```

There is room for improvement by adding a label automatically based on volumes that the agent can provide. However, if the path is used for the label, then the Flow would need to know the exact host path that would be mounted, which would couple information unnecessarily (the Flow should not know about details of the mount, only that it is provided or not):

```bash
# option b
prefect agent start docker --volume /some/path/to/x:/ctr/path
... and ...
flow.register(labels=['/some/path/to/x'], ...)
```

This could be streamlined by having the user specify the label on the agent with the volume option, though the syntax could be conflated with docker's `--volume` option, which would not behave similarly (as this is not a named volume being created):

```bash
# option c

# add a label for the volume
prefect agent start docker --volume provides-x-volume:/some/path/to/x:/ctr/path
... and ...
flow.register(labels=['provides-x-volume'], ...)

# just add the volume with no label
prefect agent start docker --volume /some/path/to/x:/ctr/path
... and ...
flow.register(...) # no label necessary
```

I'll work on implementing option a, as it is the simplest and most explicit that achieves the goal (and we would not be prevented from adding the shorthand syntax later).

(Side note outside of the scope of this issue: it could be nice to also allow for mounting of Docker named volumes for folks wanting Docker volumes without explicit host mounts... If this is the case, then the above syntax for option C would not be possible as currently expressed)
One proposed amendment to my previous comment: I'll match that the `--volume` option on an agent to match the same options as the Docker daemon. In this way users can specify rw/ro modes, named volumes, leave off container mount paths, and it will behave in the same fashion as the Docker daemon would. So these would all be valid options:
```bash
# mount host path /some/path/to/x --> container path /some/path/to/x
prefect agent start docker --volume /some/path/to/x

# mount host path /some/path/to/x --> container path /custom/location
prefect agent start docker --volume /some/path/to/x:/custom/location

# read only mount
prefect agent start docker --volume /some/path/to/x:/custom/location:ro

# named volumes
prefect agent start docker --volume a-name:/some/path/to/x
```

This implies not implementing option C would no longer be possible the above format, however, this is OK given that exposing extra options such as read-only mounts is probably more desirable (and this is behavior is probably less surprising).

Resolving patch:
diff --git a/src/prefect/agent/docker/agent.py b/src/prefect/agent/docker/agent.py
--- a/src/prefect/agent/docker/agent.py
+++ b/src/prefect/agent/docker/agent.py
@@ -1,6 +1,9 @@
+import re
 import multiprocessing
+import ntpath
+import posixpath
 from sys import platform
-from typing import TYPE_CHECKING, Iterable, List
+from typing import TYPE_CHECKING, Dict, Iterable, List, Tuple
 
 from prefect import config, context
 from prefect.agent import Agent
@@ -41,6 +44,7 @@ class DockerAgent(Agent):
             Defaults to `False` if not provided here or in context.
         - show_flow_logs (bool, optional): a boolean specifying whether the agent should re-route Flow run logs
             to stdout; defaults to `False`
+        - volumes (List[str], optional): a list of Docker volume mounts to be attached to any and all created containers.
     """
 
     def __init__(
@@ -50,6 +54,7 @@ def __init__(
         env_vars: dict = None,
         base_url: str = None,
         no_pull: bool = None,
+        volumes: List[str] = None,
         show_flow_logs: bool = False,
     ) -> None:
         super().__init__(name=name, labels=labels, env_vars=env_vars)
@@ -69,6 +74,13 @@ def __init__(
         self.no_pull = no_pull or context.get("no_pull", False)
         self.logger.debug("no_pull set to {}".format(self.no_pull))
 
+        # Resolve volumes from specs
+        (
+            self.named_volumes,
+            self.container_mount_paths,
+            self.host_spec,
+        ) = self._parse_volume_spec(volumes or [])
+
         self.failed_connections = 0
         self.docker_client = self._get_docker_client()
         self.show_flow_logs = show_flow_logs
@@ -117,6 +129,143 @@ def on_shutdown(self) -> None:
             if proc.is_alive():
                 proc.terminate()
 
+    def _is_named_volume_unix(self, canditate_path: str) -> bool:
+        if not canditate_path:
+            return False
+
+        return not canditate_path.startswith((".", "/", "~"))
+
+    def _is_named_volume_win32(self, canditate_path: str) -> bool:
+        result = self._is_named_volume_unix(canditate_path)
+
+        return (
+            result
+            and not re.match(r"^[A-Za-z]\:\\.*", canditate_path)
+            and not canditate_path.startswith("\\")
+        )
+
+    def _parse_volume_spec(
+        self, volume_specs: List[str]
+    ) -> Tuple[Iterable[str], Iterable[str], Dict[str, Dict[str, str]]]:
+        if platform == "win32":
+            return self._parse_volume_spec_win32(volume_specs)
+        return self._parse_volume_spec_unix(volume_specs)
+
+    def _parse_volume_spec_win32(
+        self, volume_specs: List[str]
+    ) -> Tuple[Iterable[str], Iterable[str], Dict[str, Dict[str, str]]]:
+        named_volumes = []  # type: List[str]
+        container_mount_paths = []  # type: List[str]
+        host_spec = {}  # type: Dict[str, Dict[str, str]]
+
+        for volume_spec in volume_specs:
+            fields = volume_spec.split(":")
+
+            if fields[-1] in ("ro", "rw"):
+                mode = fields.pop()
+            else:
+                mode = "rw"
+
+            if len(fields) == 3 and len(fields[0]) == 1:
+                # C:\path1:/path2   <-- extenal and internal path
+                external = ntpath.normpath(":".join(fields[0:2]))
+                internal = posixpath.normpath(fields[2])
+            elif len(fields) == 2:
+                combined_path = ":".join(fields)
+                (drive, path) = ntpath.splitdrive(combined_path)
+                if drive:
+                    # C:\path1          <-- assumed container path of /path1
+                    external = ntpath.normpath(combined_path)
+
+                    # C:\path1  --> /c/path1
+                    path = str("/" + drive.lower().rstrip(":") + path).replace(
+                        "\\", "/"
+                    )
+                    internal = posixpath.normpath(path)
+                else:
+                    # /path1:\path2     <-- extenal and internal path (relative to current drive)
+                    # C:/path2          <-- valid named volume
+                    external = ntpath.normpath(fields[0])
+                    internal = posixpath.normpath(fields[1])
+            elif len(fields) == 1:
+                # \path1          <-- assumed container path of /path1 (relative to current drive)
+                external = ntpath.normpath(fields[0])
+                internal = external
+            else:
+                raise ValueError(
+                    "Unable to parse volume specification '{}'".format(volume_spec)
+                )
+
+            container_mount_paths.append(internal)
+
+            if external and self._is_named_volume_win32(external):
+                named_volumes.append(external)
+                if mode != "rw":
+                    raise ValueError(
+                        "Named volumes can only have 'rw' mode, provided '{}'".format(
+                            mode
+                        )
+                    )
+            else:
+                if not external:
+                    # no internal container path given, assume the host path is the same as the internal path
+                    external = internal
+                host_spec[external] = {
+                    "bind": internal,
+                    "mode": mode,
+                }
+
+        return named_volumes, container_mount_paths, host_spec
+
+    def _parse_volume_spec_unix(
+        self, volume_specs: List[str]
+    ) -> Tuple[Iterable[str], Iterable[str], Dict[str, Dict[str, str]]]:
+        named_volumes = []  # type: List[str]
+        container_mount_paths = []  # type: List[str]
+        host_spec = {}  # type: Dict[str, Dict[str, str]]
+
+        for volume_spec in volume_specs:
+            fields = volume_spec.split(":")
+
+            if len(fields) > 3:
+                raise ValueError(
+                    "Docker volume format is invalid: {} (should be 'external:internal[:mode]')".format(
+                        volume_spec
+                    )
+                )
+
+            if len(fields) == 1:
+                external = None
+                internal = posixpath.normpath(fields[0].strip())
+            else:
+                external = posixpath.normpath(fields[0].strip())
+                internal = posixpath.normpath(fields[1].strip())
+
+            mode = "rw"
+            if len(fields) == 3:
+                mode = fields[2]
+
+            container_mount_paths.append(internal)
+
+            if external and self._is_named_volume_unix(external):
+                named_volumes.append(external)
+                if mode != "rw":
+                    raise ValueError(
+                        "Named volumes can only have 'rw' mode, provided '{}'".format(
+                            mode
+                        )
+                    )
+            else:
+                if not external:
+                    # no internal container path given, assume the host path is the same as the internal path
+                    external = internal
+                host_spec[external] = {
+                    "bind": internal,
+                    "mode": mode,
+                }
+
+        return named_volumes, container_mount_paths, host_spec
+
     def deploy_flow(self, flow_run: GraphQLResult) -> str:
         """
         Deploy flow runs on your local machine as Docker containers
@@ -134,6 +283,10 @@ def deploy_flow(self, flow_run: GraphQLResult) -> str:
             "Deploying flow run {}".format(flow_run.id)  # type: ignore
         )
 
+        # 'import docker' is expensive time-wise, we should do this just-in-time to keep
+        # the 'import prefect' time low
+        import docker
+
         storage = StorageSchema().load(flow_run.flow.storage)
         if not isinstance(StorageSchema().load(flow_run.flow.storage), Docker):
             self.logger.error(
@@ -153,10 +306,33 @@ def deploy_flow(self, flow_run: GraphQLResult) -> str:
                 self.logger.debug(line)
             self.logger.info("Successfully pulled image {}...".format(storage.name))
 
+        # Create any named volumes (if they do not already exist)
+        for named_volume_name in self.named_volumes:
+            try:
+                self.docker_client.inspect_volume(name=named_volume_name)
+            except docker.errors.APIError:
+                self.logger.debug("Creating named volume {}".format(named_volume_name))
+                self.docker_client.create_volume(
+                    name=named_volume_name,
+                    driver="local",
+                    labels={"prefect_created": "true"},
+                )
+
         # Create a container
         self.logger.debug("Creating Docker container {}".format(storage.name))
+
+        container_mount_paths = self.container_mount_paths
+        if not container_mount_paths:
+            host_config = None
+        else:
+            host_config = self.docker_client.create_host_config(binds=self.host_spec)
+
         container = self.docker_client.create_container(
-            storage.name, command="prefect execute cloud-flow", environment=env_vars
+            storage.name,
+            command="prefect execute cloud-flow",
+            environment=env_vars,
+            volumes=container_mount_paths,
+            host_config=host_config,
         )
 
         # Start the container
diff --git a/src/prefect/cli/agent.py b/src/prefect/cli/agent.py
--- a/src/prefect/cli/agent.py
+++ b/src/prefect/cli/agent.py
@@ -103,6 +103,12 @@ def agent():
     hidden=True,
 )
 @click.option("--base-url", "-b", help="Docker daemon base URL.", hidden=True)
+@click.option(
+    "--volume",
+    multiple=True,
+    help="Host paths for Docker bind mount volumes attached to each Flow runtime container.",
+    hidden=True,
+)
 @click.pass_context
 def start(
     ctx,
@@ -118,6 +124,7 @@ def start(
     base_url,
     import_path,
     show_flow_logs,
+    volume,
 ):
     """
     Start an agent.
@@ -153,6 +160,8 @@ def start(
         --base-url, -b  TEXT    A Docker daemon host URL for a DockerAgent
         --no-pull               Pull images for a DockerAgent
                                 Defaults to pulling if not provided
+        --volume        TEXT    Host paths for Docker bind mount volumes attached to each Flow runtime container.
+                                Multiple values supported e.g. `--volume /some/path --volume /some/other/path`
 
     \b
     Kubernetes Agent Options:
@@ -206,6 +215,7 @@ def start(
                 base_url=base_url,
                 no_pull=no_pull,
                 show_flow_logs=show_flow_logs,
+                volumes=list(volume),
             ).start()
         elif agent_option == "fargate":
             from_qualified_name(retrieved_agent)(


Test patch:
diff --git a/tests/agent/test_docker_agent.py b/tests/agent/test_docker_agent.py
--- a/tests/agent/test_docker_agent.py
+++ b/tests/agent/test_docker_agent.py
@@ -1,3 +1,4 @@
+import sys
 from unittest.mock import MagicMock
 
 import pytest
@@ -486,3 +487,244 @@ def test_docker_agent_heartbeat_resets_fail_count(monkeypatch, runner_token, cap
     agent.heartbeat()
     assert agent.failed_connections == 0
     assert api.ping.call_count == 4
+
+
+def test_docker_agent_init_volume_empty_options(monkeypatch, runner_token):
+    api = MagicMock()
+    monkeypatch.setattr(
+        "prefect.agent.docker.agent.DockerAgent._get_docker_client",
+        MagicMock(return_value=api),
+    )
+
+    agent = DockerAgent()
+    assert agent
+    assert agent.named_volumes == []
+    assert agent.container_mount_paths == []
+    assert agent.host_spec == {}
+
+
+@pytest.mark.parametrize(
+    "path,result",
+    [
+        ("name", True),
+        ("/some/path", False),
+        ("./some/path", False),
+        ("~/some/path", False),
+        ("../some/path", False),
+        (" ../some/path", True),  # it is up to the caller to strip the string
+        ("\n../some/path", True),  # it is up to the caller to strip the string
+    ],
+)
+def test_docker_agent_is_named_volume_unix(monkeypatch, runner_token, path, result):
+    api = MagicMock()
+    monkeypatch.setattr(
+        "prefect.agent.docker.agent.DockerAgent._get_docker_client",
+        MagicMock(return_value=api),
+    )
+    monkeypatch.setattr("prefect.agent.docker.agent.platform", "osx")
+
+    agent = DockerAgent()
+
+    assert agent._is_named_volume_unix(path) == result
+
+
+@pytest.mark.parametrize(
+    "path,result",
+    [
+        ("name", True),
+        ("C:\\\\some\\path", False),
+        ("c:\\\\some\\path", False),
+        ("\\\\some\\path", False),
+        ("\\\\\\some\\path", False),
+    ],
+)
+def test_docker_agent_is_named_volume_win32(monkeypatch, runner_token, path, result):
+    api = MagicMock()
+    monkeypatch.setattr(
+        "prefect.agent.docker.agent.DockerAgent._get_docker_client",
+        MagicMock(return_value=api),
+    )
+    monkeypatch.setattr("prefect.agent.docker.agent.platform", "win32")
+
+    agent = DockerAgent()
+
+    assert agent._is_named_volume_win32(path) == result
+
+
+@pytest.mark.parametrize(
+    "candidate,named_volumes,container_mount_paths,host_spec",
+    [
+        (
+            # handle no volume spec
+            [],
+            [],
+            [],
+            {},
+        ),
+        (
+            # no external path given (assume same as host path)
+            ["/some/path"],
+            [],
+            ["/some/path"],
+            {"/some/path": {"bind": "/some/path", "mode": "rw",}},
+        ),
+        (
+            # internal & external paths
+            ["/some/path:/ctr/path"],
+            [],
+            ["/ctr/path"],
+            {"/some/path": {"bind": "/ctr/path", "mode": "rw",}},
+        ),
+        (
+            # internal & external paths with mode
+            ["/some/path:/ctr/path:ro"],
+            [],
+            ["/ctr/path"],
+            {"/some/path": {"bind": "/ctr/path", "mode": "ro",}},
+        ),
+        (
+            # named volume
+            ["some-name:/ctr/path"],
+            ["some-name"],
+            ["/ctr/path"],
+            {},
+        ),
+        (
+            # multiple volumes
+            [
+                "some-name:/ctr/path3",
+                "/some/path:/ctr/path1",
+                "/another/path:/ctr/path2:ro",
+            ],
+            ["some-name"],
+            ["/ctr/path3", "/ctr/path1", "/ctr/path2"],
+            {
+                "/another/path": {"bind": "/ctr/path2", "mode": "ro"},
+                "/some/path": {"bind": "/ctr/path1", "mode": "rw",},
+            },
+        ),
+    ],
+)
+def test_docker_agent_parse_volume_spec_unix(
+    monkeypatch,
+    runner_token,
+    candidate,
+    named_volumes,
+    container_mount_paths,
+    host_spec,
+):
+    api = MagicMock()
+    monkeypatch.setattr(
+        "prefect.agent.docker.agent.DockerAgent._get_docker_client",
+        MagicMock(return_value=api),
+    )
+
+    agent = DockerAgent()
+
+    (
+        actual_named_volumes,
+        actual_container_mount_paths,
+        actual_host_spec,
+    ) = agent._parse_volume_spec_unix(candidate)
+
+    assert actual_named_volumes == named_volumes
+    assert actual_container_mount_paths == container_mount_paths
+    assert actual_host_spec == host_spec
+
+
+@pytest.mark.parametrize(
+    "candidate,named_volumes,container_mount_paths,host_spec",
+    [
+        (
+            # windows host --> linux container
+            ["C:\\some\\path"],
+            [],
+            ["/c/some/path"],
+            {"C:\\some\\path": {"bind": "/c/some/path", "mode": "rw",}},
+        ),
+        (
+            # internal & external paths
+            ["C:\\some\\path:/ctr/path"],
+            [],
+            ["/ctr/path"],
+            {"C:\\some\\path": {"bind": "/ctr/path", "mode": "rw",}},
+        ),
+        (
+            # internal & external paths with mode
+            ["C:\\some\\path:/ctr/path:ro"],
+            [],
+            ["/ctr/path"],
+            {"C:\\some\\path": {"bind": "/ctr/path", "mode": "ro",}},
+        ),
+        (
+            # named volume
+            ["some-name:/ctr/path"],
+            ["some-name"],
+            ["/ctr/path"],
+            {},
+        ),
+        (
+            # multiple volumes
+            [
+                "some-name:/ctr/path3",
+                "C:\\some\\path:/ctr/path1",
+                "D:\\another\\path:/ctr/path2:ro",
+            ],
+            ["some-name"],
+            ["/ctr/path3", "/ctr/path1", "/ctr/path2"],
+            {
+                "D:\\another\\path": {"bind": "/ctr/path2", "mode": "ro"},
+                "C:\\some\\path": {"bind": "/ctr/path1", "mode": "rw",},
+            },
+        ),
+    ],
+)
+def test_docker_agent_parse_volume_spec_win(
+    monkeypatch,
+    runner_token,
+    candidate,
+    named_volumes,
+    container_mount_paths,
+    host_spec,
+):
+    api = MagicMock()
+    monkeypatch.setattr(
+        "prefect.agent.docker.agent.DockerAgent._get_docker_client",
+        MagicMock(return_value=api),
+    )
+
+    agent = DockerAgent()
+
+    (
+        actual_named_volumes,
+        actual_container_mount_paths,
+        actual_host_spec,
+    ) = agent._parse_volume_spec_win32(candidate)
+
+    assert actual_named_volumes == named_volumes
+    assert actual_container_mount_paths == container_mount_paths
+    assert actual_host_spec == host_spec
+
+
+@pytest.mark.parametrize(
+    "candidate,exception_type",
+    [
+        # named volumes cannot be read only
+        ("some-name:/ctr/path:ro", ValueError),
+        # dont attempt to parse too many fields
+        ("/some/path:/ctr/path:rw:something-else", ValueError),
+    ],
+)
+def test_docker_agent_parse_volume_spec_raises_on_invalid_spec(
+    monkeypatch, runner_token, candidate, exception_type,
+):
+    api = MagicMock()
+    monkeypatch.setattr(
+        "prefect.agent.docker.agent.DockerAgent._get_docker_client",
+        MagicMock(return_value=api),
+    )
+
+    agent = DockerAgent()
+
+    with pytest.raises(exception_type):
+        agent._parse_volume_spec([candidate])
diff --git a/tests/cli/test_agent.py b/tests/cli/test_agent.py
--- a/tests/cli/test_agent.py
+++ b/tests/cli/test_agent.py
@@ -195,6 +195,7 @@ def test_agent_start_with_env_vars(monkeypatch, runner_token):
         name=None,
         no_pull=False,
         show_flow_logs=False,
+        volumes=[],
     )