Repository: PrefectHQ/prefect

Issue:
Expose ability to set location and / or  permissions on Flow files within Docker storage
Currently, we place Flow objects in the `/root` directory of their Docker containers.  This can be a problem when root access is somehow restricted.  We should introduce two new APIs:

- introduce a way to configure the directory / location of the flow within the image
- optionally set the correct permissions on the flow file so that non-root users can read / write it


Discussion:
OpenShift Container Platform, which runs containers using an `arbitrarily assigned user ID` (i.e. non-root). This provides additional security against processes escaping the container due to a container engine vulnerability and thereby achieving escalated permissions on the host node.

To clarify on the problem, when the `COPY` command is used within the DockerFile, it will copy the file into the Docker Image with the exact user permission mask as it was created on the host system generating the Docker Image.

In the case of Prefect, when registering a Flow and opting to have Prefect build the Docker Image, the code will attempt to cloudpickle the Flow into a temporary directory on the system performing the build, then `COPY` that file into the Docker Image.
https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/storage/docker.py#L420

Setting the `umask` to override the `COPY` command within the Docker Daemon is not feasible, so the only workaround we have today is to set the `umask 002` on the host system performing the Prefect `register` command. This allows you to specify the default file permission on the host system, before it's `COPY` into the Docker Image. Then once it's in the Docker Image, it'll have the appropriate permissions set on the host system. 

When registering Flows on a local PC, it's pretty easy to modify the `umask 002` (at least on Mac). However, when working with shared CI/CD tools, we don't always have control over specifying the `umask` on the machine which will perform the `register`.

It'd be great if the code could be modified, such that after the `COPY` command, it would update the file permissions to allow the `root` group to have rwX permissions (`chgrp -R 0  {dest} && chmod -R g+rwX {dest}`). This would have zero impact on normal flows, and it would then enable non-root execution of Prefect Flows.
Actually, the `umask` trick doesn't work in this context. Since Prefect is using the Python open to dump the file, it avoids using the `umask` and explicitly sets 644 as the permissions on the *.flow files cloudpickle is creating.

After it creates the flow file on disk, you could chmod there to allow the `root` group `write` access. Which would avoid having to run the chgrp and chmod commands within another Docker file layer:
```python
os.chmod(flow_path, 0o664)
```
From discussion with the core team, it sounds like there is also an option here (which introduces less configuration bloat which we liked) for us to instead store at something like `/opt/prefect/flows`. Since Cloud knows the absolute path this shouldn't affect any users to change this way. Fundamentally the important part about having used `/root` in the first place was just that we could guarantee that directory would be available but if we can guarantee `/opt/prefect/flows` from the Dockerfile command we feel this is good. 

Resolving patch:
diff --git a/src/prefect/environments/storage/docker.py b/src/prefect/environments/storage/docker.py
--- a/src/prefect/environments/storage/docker.py
+++ b/src/prefect/environments/storage/docker.py
@@ -105,7 +105,7 @@ def __init__(
 
         self.env_vars = env_vars or {}
         self.env_vars.setdefault(
-            "PREFECT__USER_CONFIG_PATH", "/root/.prefect/config.toml"
+            "PREFECT__USER_CONFIG_PATH", "/opt/prefect/config.toml"
         )
 
         self.files = files or {}
@@ -219,7 +219,7 @@ def add_flow(self, flow: "prefect.core.flow.Flow") -> str:
                     flow.name
                 )
             )
-        flow_path = "/root/.prefect/flows/{}.prefect".format(slugify(flow.name))
+        flow_path = "/opt/prefect/flows/{}.prefect".format(slugify(flow.name))
         self.flows[flow.name] = flow_path
         self._flows[flow.name] = flow  # needed prior to build
         return flow_path
@@ -459,9 +459,9 @@ def create_dockerfile_object(self, directory: str) -> str:
             {extra_commands}
             {pip_installs}
 
-            RUN mkdir -p /root/.prefect/
+            RUN mkdir -p /opt/prefect/
             {copy_flows}
-            COPY {healthcheck_loc} /root/.prefect/healthcheck.py
+            COPY {healthcheck_loc} /opt/prefect/healthcheck.py
             {copy_files}
 
             {env_vars}
@@ -483,7 +483,7 @@ def create_dockerfile_object(self, directory: str) -> str:
             file_contents += textwrap.dedent(
                 """
 
-                RUN python /root/.prefect/healthcheck.py '[{flow_file_paths}]' '{python_version}'
+                RUN python /opt/prefect/healthcheck.py '[{flow_file_paths}]' '{python_version}'
                 """.format(
                     flow_file_paths=", ".join(
                         ['"{}"'.format(k) for k in self.flows.values()]


Test patch:
diff --git a/tests/environments/storage/test_docker_storage.py b/tests/environments/storage/test_docker_storage.py
--- a/tests/environments/storage/test_docker_storage.py
+++ b/tests/environments/storage/test_docker_storage.py
@@ -37,9 +37,9 @@ def test_add_flow_to_docker():
     storage = Docker()
     f = Flow("test")
     assert f not in storage
-    assert storage.add_flow(f) == "/root/.prefect/flows/test.prefect"
+    assert storage.add_flow(f) == "/opt/prefect/flows/test.prefect"
     assert f.name in storage
-    assert storage.flows[f.name] == "/root/.prefect/flows/test.prefect"
+    assert storage.flows[f.name] == "/opt/prefect/flows/test.prefect"
 
 
 @pytest.mark.parametrize(
@@ -61,9 +61,7 @@ def test_empty_docker_storage(monkeypatch, platform, url):
     assert not storage.image_name
     assert not storage.image_tag
     assert storage.python_dependencies == ["wheel"]
-    assert storage.env_vars == {
-        "PREFECT__USER_CONFIG_PATH": "/root/.prefect/config.toml"
-    }
+    assert storage.env_vars == {"PREFECT__USER_CONFIG_PATH": "/opt/prefect/config.toml"}
     assert not storage.files
     assert storage.prefect_version
     assert storage.base_url == url
@@ -90,9 +88,7 @@ def test_empty_docker_storage_on_tagged_commit(monkeypatch, platform, url):
     assert not storage.image_name
     assert not storage.image_tag
     assert storage.python_dependencies == ["wheel"]
-    assert storage.env_vars == {
-        "PREFECT__USER_CONFIG_PATH": "/root/.prefect/config.toml"
-    }
+    assert storage.env_vars == {"PREFECT__USER_CONFIG_PATH": "/opt/prefect/config.toml"}
     assert not storage.files
     assert storage.prefect_version
     assert storage.base_url == url
@@ -142,7 +138,7 @@ def test_initialized_docker_storage():
     assert storage.python_dependencies == ["test", "wheel"]
     assert storage.env_vars == {
         "test": "1",
-        "PREFECT__USER_CONFIG_PATH": "/root/.prefect/config.toml",
+        "PREFECT__USER_CONFIG_PATH": "/opt/prefect/config.toml",
     }
     assert storage.base_url == "test_url"
     assert storage.prefect_version == "my-branch"
@@ -396,7 +392,7 @@ def test_create_dockerfile_from_dockerfile_uses_tempdir_path():
                 output = dockerfile.read()
 
             assert (
-                "COPY {} /root/.prefect/flows/foo.prefect".format(
+                "COPY {} /opt/prefect/flows/foo.prefect".format(
                     os.path.join(directory, "foo.flow")
                 )
                 in output
@@ -405,7 +401,7 @@ def test_create_dockerfile_from_dockerfile_uses_tempdir_path():
                 "COPY {} ./test2".format(os.path.join(directory, "test")) in output
             ), output
             assert (
-                "COPY {} /root/.prefect/healthcheck.py".format(
+                "COPY {} /opt/prefect/healthcheck.py".format(
                     os.path.join(directory, "healthcheck.py")
                 )
                 in output
@@ -472,7 +468,7 @@ def test_create_dockerfile_with_weird_flow_name():
                 output = dockerfile.read()
 
             assert (
-                "COPY what-is-this.flow /root/.prefect/flows/what-is-this.prefect"
+                "COPY what-is-this.flow /opt/prefect/flows/what-is-this.prefect"
                 in output
             )
 
@@ -517,9 +513,9 @@ def test_create_dockerfile_from_everything():
             assert results != None
             assert results.group("result") == "test=1"
 
-            assert "COPY healthcheck.py /root/.prefect/healthcheck.py" in output
-            assert "COPY test.flow /root/.prefect/flows/test.prefect" in output
-            assert "COPY other.flow /root/.prefect/flows/other.prefect" in output
+            assert "COPY healthcheck.py /opt/prefect/healthcheck.py" in output
+            assert "COPY test.flow /opt/prefect/flows/test.prefect" in output
+            assert "COPY other.flow /opt/prefect/flows/other.prefect" in output
 
 
 @pytest.mark.parametrize("ignore_healthchecks", [True, False])
@@ -541,9 +537,9 @@ def test_run_healthchecks_arg(ignore_healthchecks):
                 output = dockerfile.read()
 
             if ignore_healthchecks:
-                assert "RUN python /root/.prefect/healthcheck.py" not in output
+                assert "RUN python /opt/prefect/healthcheck.py" not in output
             else:
-                assert "RUN python /root/.prefect/healthcheck.py" in output
+                assert "RUN python /opt/prefect/healthcheck.py" in output
 
 
 def test_pull_image(capsys, monkeypatch):
diff --git a/tests/serialization/test_storage.py b/tests/serialization/test_storage.py
--- a/tests/serialization/test_storage.py
+++ b/tests/serialization/test_storage.py
@@ -70,7 +70,7 @@ def test_docker_serialize_with_flows():
     assert serialized["image_name"] == "name"
     assert serialized["image_tag"] == "tag"
     assert serialized["registry_url"] == "url"
-    assert serialized["flows"] == {"test": "/root/.prefect/flows/test.prefect"}
+    assert serialized["flows"] == {"test": "/opt/prefect/flows/test.prefect"}
     assert serialized["secrets"] == ["FOO"]
 
     deserialized = DockerSchema().load(serialized)