Repository: PrefectHQ/prefect

Issue:
Make prefect directory inside Docker storage configurable
## Current behavior

Today when you build a `Docker` storage, `prefect` creates a directory for itself called `/opt/prefect` inside the image.

https://github.com/PrefectHQ/prefect/blob/467bc5b1dcd83716bd896eff549f6bceb59da8cf/src/prefect/environments/storage/docker.py#L469

That directory is used to store configuration files and flows.

## Proposed behavior

I'd like to propose making `prefect`'s directory inside `Docker` storage configurable. Ideally I'd like to be able to do this or something similar:

```python
flow.storage = Docker(
    prefect_directory="/home/my-custom-user/prefect"
)
```

If this is picked up, I believe it would mean the following:

* add a new keyword argument to `Docker` (https://github.com/PrefectHQ/prefect/blob/467bc5b1dcd83716bd896eff549f6bceb59da8cf/src/prefect/environments/storage/docker.py#L81), which defaults to `/opt/prefect` for backwards compatibility
* replace all hard-coded `/opt/prefect` in https://github.com/PrefectHQ/prefect/blob/467bc5b1dcd83716bd896eff549f6bceb59da8cf/src/prefect/environments/storage/docker.py with a reference to that attribute, `self.prefect_directory`

## Example

The current state can lead to issues if the `base_image` you're using runs as a user that does not have permission to write to `/opt`. I just encountered this situation using a base_image where the default user cannot write to `/opt`

> Step 5/9 : RUN mkdir -p /opt/prefect/
 ---> Running in eb30b890ad2b
 mkdir: cannot create directory ‘/opt/prefect/’: Permission denied

Making this configurable could remove an existing limitation that might currently lead people to run the image build process as `root`.

## Notes for Reviewers

If the maintainers here are interested in this change, I'd be happy to contribute it.

Thanks for your time and consideration!


Discussion:
> Making this configurable could remove an existing limitation that might currently lead people to run the image build process as root.

When I build images, I generally have the build run as root, and use a different uid at runtime. This avoids issues at build time, but still keeps you secure at runtime. I can't think of a security issue for using root at buildtime.

That said, I would be accept making this configurable.
@jcrist , makes sense to me! I agree with you, but I'm working with base images that I don't totally control that make use of [`USER`](https://docs.docker.com/engine/reference/builder/#user).

I'll submit a PR to make this configurable, thanks!

Resolving patch:
diff --git a/src/prefect/environments/storage/docker.py b/src/prefect/environments/storage/docker.py
--- a/src/prefect/environments/storage/docker.py
+++ b/src/prefect/environments/storage/docker.py
@@ -73,6 +73,8 @@ class Docker(Storage):
         - build_kwargs (dict, optional): Additional keyword arguments to pass to Docker's build
             step. [Documentation](https://docker-py.readthedocs.io/en/stable/
             api.html#docker.api.build.BuildApiMixin.build)
+        - prefect_directory (str, optional): Path to the directory where prefect configuration/flows
+             should be stored inside the Docker image. Defaults to `/opt/prefect`.
         - **kwargs (Any, optional): any additional `Storage` initialization options
 
     Raises:
@@ -96,6 +98,7 @@ def __init__(
         base_url: str = None,
         tls_config: Union[bool, "docker.tls.TLSConfig"] = False,
         build_kwargs: dict = None,
+        prefect_directory: str = "/opt/prefect",
         **kwargs: Any,
     ) -> None:
         self.registry_url = registry_url
@@ -108,9 +111,11 @@ def __init__(
         self.python_dependencies = python_dependencies or []
         self.python_dependencies.append("wheel")
 
+        self.prefect_directory = prefect_directory
+
         self.env_vars = env_vars or {}
         self.env_vars.setdefault(
-            "PREFECT__USER_CONFIG_PATH", "/opt/prefect/config.toml"
+            "PREFECT__USER_CONFIG_PATH", "{}/config.toml".format(self.prefect_directory)
         )
 
         self.files = files or {}
@@ -229,7 +234,9 @@ def add_flow(self, flow: "prefect.core.flow.Flow") -> str:
                     flow.name
                 )
             )
-        flow_path = "/opt/prefect/flows/{}.prefect".format(slugify(flow.name))
+        flow_path = "{}/flows/{}.prefect".format(
+            self.prefect_directory, slugify(flow.name)
+        )
         self.flows[flow.name] = flow_path
         self._flows[flow.name] = flow  # needed prior to build
         return flow_path
@@ -470,9 +477,9 @@ def create_dockerfile_object(self, directory: str) -> str:
             {extra_commands}
             {pip_installs}
 
-            RUN mkdir -p /opt/prefect/
+            RUN mkdir -p {prefect_dir}/
             {copy_flows}
-            COPY {healthcheck_loc} /opt/prefect/healthcheck.py
+            COPY {healthcheck_loc} {prefect_dir}/healthcheck.py
             {copy_files}
 
             {env_vars}
@@ -486,6 +493,7 @@ def create_dockerfile_object(self, directory: str) -> str:
                 else "healthcheck.py",
                 copy_files=copy_files,
                 env_vars=env_vars,
+                prefect_dir=self.prefect_directory,
             )
         )
 
@@ -494,12 +502,13 @@ def create_dockerfile_object(self, directory: str) -> str:
             file_contents += textwrap.dedent(
                 """
 
-                RUN python /opt/prefect/healthcheck.py '[{flow_file_paths}]' '{python_version}'
+                RUN python {prefect_dir}/healthcheck.py '[{flow_file_paths}]' '{python_version}'
                 """.format(
                     flow_file_paths=", ".join(
                         ['"{}"'.format(k) for k in self.flows.values()]
                     ),
                     python_version=(sys.version_info.major, sys.version_info.minor),
+                    prefect_dir=self.prefect_directory,
                 )
             )
 


Test patch:
diff --git a/tests/environments/storage/test_docker_storage.py b/tests/environments/storage/test_docker_storage.py
--- a/tests/environments/storage/test_docker_storage.py
+++ b/tests/environments/storage/test_docker_storage.py
@@ -51,6 +51,18 @@ def test_add_flow_to_docker():
     assert storage.flows[f.name] == "/opt/prefect/flows/test.prefect"
 
 
+def test_add_flow_to_docker_custom_prefect_dir():
+    storage = Docker(prefect_directory="/usr/prefect-is-gr8")
+    f = Flow("test")
+    assert f not in storage
+    assert storage.add_flow(f) == "/usr/prefect-is-gr8/flows/test.prefect"
+    assert f.name in storage
+    assert storage.flows[f.name] == "/usr/prefect-is-gr8/flows/test.prefect"
+    assert storage.env_vars == {
+        "PREFECT__USER_CONFIG_PATH": "/usr/prefect-is-gr8/config.toml"
+    }
+
+
 @pytest.mark.parametrize(
     "platform,url",
     [
@@ -515,6 +527,29 @@ def test_create_dockerfile_with_weird_flow_name():
             )
 
 
+def test_create_dockerfile_with_weird_flow_name_custom_prefect_dir(tmpdir):
+
+    with open(os.path.join(tmpdir, "test"), "w+") as t:
+        t.write("asdf")
+
+    storage = Docker(
+        registry_url="test1",
+        base_image="test3",
+        prefect_directory="/tmp/prefect-is-c00l",
+    )
+    f = Flow("WHAT IS THIS !!! ~~~~")
+    storage.add_flow(f)
+    dpath = storage.create_dockerfile_object(directory=tmpdir)
+
+    with open(dpath, "r") as dockerfile:
+        output = dockerfile.read()
+
+    assert (
+        "COPY what-is-this.flow /tmp/prefect-is-c00l/flows/what-is-this.prefect"
+        in output
+    )
+
+
 def test_create_dockerfile_from_everything(no_docker_host_var):
 
     with tempfile.TemporaryDirectory() as tempdir_outside:
@@ -584,6 +619,29 @@ def test_run_healthchecks_arg(ignore_healthchecks):
                 assert "RUN python /opt/prefect/healthcheck.py" in output
 
 
+@pytest.mark.parametrize("ignore_healthchecks", [True, False])
+def test_run_healthchecks_arg_custom_prefect_dir(ignore_healthchecks, tmpdir):
+
+    with open(os.path.join(tmpdir, "test"), "w+") as t:
+        t.write("asdf")
+
+    storage = Docker(
+        ignore_healthchecks=ignore_healthchecks, prefect_directory="/usr/local/prefect",
+    )
+
+    f = Flow("test")
+    storage.add_flow(f)
+    dpath = storage.create_dockerfile_object(directory=tmpdir)
+
+    with open(dpath, "r") as dockerfile:
+        output = dockerfile.read()
+
+    if ignore_healthchecks:
+        assert "RUN python /usr/local/prefect/healthcheck.py" not in output
+    else:
+        assert "RUN python /usr/local/prefect/healthcheck.py" in output
+
+
 def test_pull_image(capsys, monkeypatch):
     storage = Docker(base_image="python:3.6")
 
diff --git a/tests/serialization/test_storage.py b/tests/serialization/test_storage.py
--- a/tests/serialization/test_storage.py
+++ b/tests/serialization/test_storage.py
@@ -81,6 +81,31 @@ def test_docker_serialize_with_flows():
     assert deserialized.secrets == ["FOO"]
 
 
+def test_docker_serialize_with_flow_and_custom_prefect_dir():
+    docker = storage.Docker(
+        registry_url="url",
+        image_name="name",
+        image_tag="tag",
+        secrets=["FOO"],
+        prefect_directory="/tmp/something",
+    )
+    f = prefect.Flow("test")
+    docker.add_flow(f)
+    serialized = DockerSchema().dump(docker)
+
+    assert serialized
+    assert serialized["__version__"] == prefect.__version__
+    assert serialized["image_name"] == "name"
+    assert serialized["image_tag"] == "tag"
+    assert serialized["registry_url"] == "url"
+    assert serialized["flows"] == {"test": "/tmp/something/flows/test.prefect"}
+    assert serialized["secrets"] == ["FOO"]
+
+    deserialized = DockerSchema().load(serialized)
+    assert f.name in deserialized
+    assert deserialized.secrets == ["FOO"]
+
+
 def test_s3_empty_serialize():
     s3 = storage.S3(bucket="bucket")
     serialized = S3Schema().dump(s3)