Repository: PrefectHQ/prefect

Issue:
Add option to `Docker` storage to respect `.dockerignore` file if using `files` argument
## Opened from the [Prefect Public Slack Community](https://prefect.io/slack)

**michael.ball**: Hey there. I’ve been playing around with Docker storage today, trying to get all source code packaged together with the flows each time they are registered, and am using the `files` and `env_vars`  attributes as outlined in the Docs. But it seems that my `.dockerignore` file (in the directory from which I am registering the flow) is ignored by this build process. I have massive data directories nested at different levels inside the codebase so it’s essential there be a way to include this somehow. Anyone come across this / have any ideas? I’m curious about playing around with a custom Dockerfile, but the code that copies flows to a tmp Dockerfile in `create_docker_file` is dynamic so it seems tough that I could write a standalone Dockerfile to handle this case.

**kevin701**: Hey <@U02CELMJ5EJ>, this is the first I’ve seen this. I’ll look into it.

**zach555**: Hey <@U02CELMJ5EJ>, can you try including `.dockerignore` in the `files` arg for the storage?

**michael.ball**: Interesting… I’ll check it out :slightly_smiling_face:

**michael.ball**: Actually wait, the `.dockerignore` is already there because it falls within the directory I’m trying to copy. I just tried to explicitly copy it as a separate `files` arg, but same deal.

**michael.ball**: It’s actually a bit strange… the `create_docker_file` outputted script just has a standard COPY command in it, which I thought was supposed to obey `.dockerignore` … Perhaps the build context is not where I think it is

**zach555**: &gt; Perhaps the build context is not where I think it is
Yes! (unless you've looked through the internals of Docker storage)

When building Docker storage, Prefect creates a temporary directory that contains a new Dockerfile (created by Prefect) and also copies over all your files. For the `.dockerignore` to be respected, it will need to be copied into the root of this temporary directory <https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/docker.py#L435>

**zach555**: But I'm not sure how exactly to do that :thinking_face:

**michael.ball**: Ah well, there’s the answer! But yes I’m also not immediately sure how to handle that nicely. I think that a .dockerignore path to go with the dockerfile one would be cool :shrug: but it looks unfixable without changing the Prefect source

**zach555**: Yeah I can open an issue for this regardless of workarounds
will it work if you specify `files` like so?

```
files = {
    ...
    "/path/to/.dockerignore": ".dockerignore"
}
```


**michael.ball**: Unfortunately not, because that just creates a COPY statement inside the generated `Dockerfile` that copies the `.dockerignore` inside the image, whereas the docker build process needs to read the `.dockerignore` file before starting to build (at least, this is how I understand why it doesn’t work)

**michael.ball**: I might be able to work around it! I saw that if I specify a Dockerfile then it uses the current directory as build context, and in that case Docker would find the `.dockerignore` file automatically

**michael.ball**: Will return to it tomorrow and lyk! Thanks for your help

**zach555**: <@ULVA73B9P> open "Add option to `Docker` storage to respect `.dockerignore` file if using `files` argument"

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1630600807008500?thread_ts=1630600807.008500&cid=CL09KU1K7).




Discussion:

Resolving patch:
diff --git a/src/prefect/storage/docker.py b/src/prefect/storage/docker.py
--- a/src/prefect/storage/docker.py
+++ b/src/prefect/storage/docker.py
@@ -8,7 +8,7 @@
 import textwrap
 import uuid
 import warnings
-from pathlib import PurePosixPath
+from pathlib import PurePosixPath, Path
 from typing import TYPE_CHECKING, Any, Iterable, List, Union
 
 import pendulum
@@ -88,6 +88,9 @@ class Docker(Storage):
             use when building
         - files (dict, optional): a dictionary of files or directories to copy into
             the image when building. Takes the format of `{'src': 'dest'}`
+        - dockerignore (str, optional): an optional Path to a `dockerignore` file.
+            when used with the `files` argument, the specified `dockerignore` will be
+            included in the build context
         - prefect_version (str, optional): an optional branch, tag, or commit
             specifying the version of prefect you want installed into the container;
             defaults to the version you are currently using or `"master"` if your
@@ -126,6 +129,7 @@ def __init__(
         registry_url: str = None,
         base_image: str = None,
         dockerfile: str = None,
+        dockerignore: str = None,
         python_dependencies: List[str] = None,
         image_name: str = None,
         image_tag: str = None,
@@ -199,6 +203,7 @@ def __init__(
             self.base_image = base_image  # type: ignore
 
         self.dockerfile = dockerfile
+        self.dockerignore = dockerignore
         # we should always try to install prefect, unless it is already installed. We can't
         # determine this until image build time.
         self.installation_commands.append(
@@ -327,7 +332,6 @@ def _build_image(self, push: bool = True) -> tuple:
         with tempfile.TemporaryDirectory(
             dir="." if self.dockerfile else None
         ) as tempdir:
-
             # Build the dockerfile
             if self.base_image and not self.local_image:
                 self.pull_image()
@@ -433,6 +437,12 @@ def create_dockerfile_object(self, directory: str) -> str:
         # Copy user specified files into the image
         copy_files = ""
         if self.files:
+            # copy dockerignore if provided
+            if self.dockerignore:
+                shutil.copyfile(
+                    src=Path(self.dockerignore).expanduser().resolve(),
+                    dst=Path(directory) / ".dockerignore",
+                )
             for src, dest in self.files.items():
                 fname = os.path.basename(src)
                 full_fname = os.path.join(directory, fname)


Test patch:
diff --git a/tests/storage/test_docker_storage.py b/tests/storage/test_docker_storage.py
--- a/tests/storage/test_docker_storage.py
+++ b/tests/storage/test_docker_storage.py
@@ -448,6 +448,36 @@ def test_copy_files():
             ), output
 
 
+def test_copy_files_with_dockerignore():
+    with tempfile.TemporaryDirectory() as sample_top_directory:
+
+        sample_sub_directory = os.path.join(sample_top_directory, "subdir")
+        os.mkdir(sample_sub_directory)
+
+        sample_file = os.path.join(sample_sub_directory, "test.txt")
+        with open(sample_file, "w+") as t:
+            t.write("asdf")
+
+        dockerignore = os.path.join(sample_sub_directory, ".dockerignore")
+        with open(dockerignore, "w+") as t:
+            t.write("test.txt")
+
+        with tempfile.TemporaryDirectory() as directory:
+
+            storage = Docker(
+                files={
+                    sample_sub_directory: "/test_dir",
+                    sample_file: "/path/test_file.txt",
+                },
+                dockerignore=dockerignore,
+            )
+            storage.add_flow(Flow("foo"))
+            storage.create_dockerfile_object(directory=directory)
+
+            contents = os.listdir(directory)
+            assert ".dockerignore" in contents, contents
+
+
 def test_extra_dockerfile_commands():
     with tempfile.TemporaryDirectory() as directory: