Repository: PrefectHQ/prefect

Issue:
Deployments built on Windows and uploaded to GCS cause runtime errors due to `/` and `\` separators
## Opened from the [Prefect Public Slack Community](https://prefect.io/slack)

**kevin.grismore065**: having trouble running my gcs-stored flows on kubernetes. I feel like it probably has something to do with how my project is structured:

```
- project
        └── flows
            └── flow1.py
            └── flow2.py
        └── util
            └── util.py
```

if I do `some/dir/project&gt; prefect deployment build flows/flow1.py:flow_func -n my-flow -ib kubernetes-job/my-job -sb gcs/my-bucket -t k8s` everything in src ends up in my bucket as expected, but when I run the flow I get:

```
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows/flow1.py'
```


**anna**: A recipe for baking flow code into a docker image is on our radar and should be released within the next 2 weeks or so

**crandall**: oooh, how serendipitous, this will be useful for me too!

**kevin.grismore065**: so, I got around this by changing my project structure.

```
- project
        └── flow1.py
        └── flow2.py
        └── util
            └── __init__.py
            └── util.py
```


**kevin.grismore065**: but I still get `ModuleNotFoundError: No module named 'util'`, or the same error for _anything_ that's in a folder as part of what gets packaged with my flow in the gcs bucket, despite including `init` files and everything running fine locally and on a local agent

**kevin.grismore065**: actually, wait... I think I might know the issue. will come back to this if I'm right. I do this a lot lol

**kevin.grismore065**: okay, _soooooo_ I think it's because I'm running the `deployment build` command on windows. when all the files are sent to my gcs bucket via `shutil.copytree()` , all the nested directories get flattened in the bucket because it doesn't recognize the `\` as a folder separator. Consequently, all the files get downloaded to the target execution environment as a flat structure too.

```
flow1.py
flow2.py
util
    └── __init__.py
    └── util.py
```

becomes

```
flow1.py
flow2.py
util\__init__.py
util\util.py
```

which... well of course python isn't going to find the modules it's trying to import

**kevin.grismore065**: this behavior looks like it's built directly in to `shutil.copytree()` - it uses `os.scandir()`. Even if you were to posix-ify the the base path, all the children will still have windows-style `\` separators:

```
&gt;&gt;&gt; path = Path('C:\Windows').as_posix()
&gt;&gt;&gt; itr = os.scandir(path)
&gt;&gt;&gt; [print(path.path) for path in itr]
C:/Windows\addins
C:/Windows\appcompat
C:/Windows\apppatch
C:/Windows\AppReadiness
```


**anna**: thanks for this thorough investigation. Let me open an issue

<@ULVA73B9P> open "Deployments built on Windows and uploaded to GCS cause runtime errors due to `/` and `\` separators"

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1659734281715809?thread_ts=1659734281.715809&cid=CL09KU1K7).




Discussion:
@anna-geller is the recipe you mention created yet? :-)
> A recipe for baking flow code into a docker image is on our radar and should be released within the next 2 weeks or so
I could already create it but it would be hacky. We decided to address it on the product side and add a feature that will handle that automatically. We plan to release that by the end of this week.
@anna-geller no need for a quick fix - I'll wait :-) 
I guess it is the same as https://github.com/PrefectHQ/prefect/issues/6175 
yup!

Resolving patch:
diff --git a/src/prefect/filesystems.py b/src/prefect/filesystems.py
--- a/src/prefect/filesystems.py
+++ b/src/prefect/filesystems.py
@@ -2,11 +2,12 @@
 import glob
 import io
 import json
+import os
 import shutil
 import sys
 import tempfile
 import urllib.parse
-from pathlib import Path
+from pathlib import Path, PurePath
 from typing import Any, Dict, Optional
 
 import anyio
@@ -319,16 +320,21 @@ async def put_directory(
             with open(ignore_file, "r") as f:
                 ignore_patterns = f.readlines()
 
-            included_files = filter_files(local_path, ignore_patterns)
+            included_files = filter_files(
+                local_path, ignore_patterns, include_dirs=True
+            )
 
         counter = 0
-        for f in glob.glob("**", recursive=True):
-            if ignore_file and f not in included_files:
+        for f in glob.glob(os.path.join(local_path, "**"), recursive=True):
+            relative_path = PurePath(f).relative_to(local_path).as_posix()
+            if included_files and relative_path not in included_files:
                 continue
+
             if to_path.endswith("/"):
-                fpath = to_path + f
+                fpath = to_path + relative_path
             else:
-                fpath = to_path + "/" + f
+                fpath = to_path + "/" + relative_path
+
             if Path(f).is_dir():
                 pass
             else:
@@ -336,7 +342,9 @@ async def put_directory(
                     self.filesystem.put_file(f, fpath, overwrite=True)
                 else:
                     self.filesystem.put_file(f, fpath)
+
             counter += 1
+
         return counter
 
     async def read_path(self, path: str) -> bytes:
@@ -444,7 +452,7 @@ async def put_directory(
         ignore_file: Optional[str] = None,
     ) -> int:
         """
-        Uploads a directory from a given local path to a remote direcotry.
+        Uploads a directory from a given local path to a remote directory.
 
         Defaults to uploading the entire contents of the current working directory to the block's basepath.
         """
diff --git a/src/prefect/utilities/filesystem.py b/src/prefect/utilities/filesystem.py
--- a/src/prefect/utilities/filesystem.py
+++ b/src/prefect/utilities/filesystem.py
@@ -28,7 +28,7 @@ def set_default_ignore_file(path: str) -> bool:
 
 def filter_files(
     root: str = ".", ignore_patterns: list = None, include_dirs: bool = True
-) -> list:
+) -> set:
     """
     This function accepts a root directory path and a list of file patterns to ignore, and returns
     a list of files that excludes those that should be ignored.


Test patch:
diff --git a/tests/test-projects/flat-project/.prefectignore b/tests/test-projects/flat-project/.prefectignore
new file mode 100644
--- /dev/null
+++ b/tests/test-projects/flat-project/.prefectignore
@@ -0,0 +1,2 @@
+.prefectignore
+__pycache__
diff --git a/tests/test-projects/tree-project/.prefectignore b/tests/test-projects/tree-project/.prefectignore
new file mode 100644
--- /dev/null
+++ b/tests/test-projects/tree-project/.prefectignore
@@ -0,0 +1,2 @@
+.prefectignore
+__pycache__/
diff --git a/tests/test_filesystems.py b/tests/test_filesystems.py
--- a/tests/test_filesystems.py
+++ b/tests/test_filesystems.py
@@ -1,3 +1,4 @@
+import os
 from pathlib import Path
 
 import pytest
@@ -6,6 +7,8 @@
 from prefect.filesystems import GitHub, LocalFileSystem, RemoteFileSystem
 from prefect.testing.utilities import AsyncMock
 
+TEST_PROJECTS_DIR = prefect.__root_path__ / "tests" / "test-projects"
+
 
 class TestLocalFileSystem:
     async def test_read_write_roundtrip(self, tmp_path):
@@ -90,6 +93,41 @@ async def test_resolve_path(self):
         assert fs._resolve_path(f"{base}/subdir") == f"{base}/subdir"
         assert fs._resolve_path("subdirectory") == f"{base}/subdirectory"
 
+    async def test_put_directory_flat(self):
+        fs = RemoteFileSystem(basepath="memory://flat")
+        await fs.put_directory(
+            os.path.join(TEST_PROJECTS_DIR, "flat-project"),
+            ignore_file=os.path.join(
+                TEST_PROJECTS_DIR, "flat-project", ".prefectignore"
+            ),
+        )
+        copied_files = set(fs.filesystem.glob("/flat/**"))
+
+        assert copied_files == {
+            "/flat/explicit_relative.py",
+            "/flat/implicit_relative.py",
+            "/flat/shared_libs.py",
+        }
+
+    async def test_put_directory_tree(self):
+        fs = RemoteFileSystem(basepath="memory://tree")
+        await fs.put_directory(
+            os.path.join(TEST_PROJECTS_DIR, "tree-project"),
+            ignore_file=os.path.join(
+                TEST_PROJECTS_DIR, "tree-project", ".prefectignore"
+            ),
+        )
+        copied_files = set(fs.filesystem.glob("/tree/**"))
+
+        assert copied_files == {
+            "/tree/imports",
+            "/tree/imports/explicit_relative.py",
+            "/tree/imports/implicit_relative.py",
+            "/tree/shared_libs",
+            "/tree/shared_libs/bar.py",
+            "/tree/shared_libs/foo.py",
+        }
+
 
 class TestGitHub:
     async def test_subprocess_errors_are_surfaced(self):