Repository: PrefectHQ/prefect

Issue:
Cloudpickle error when base_image isn't specified
## Description
If you remove the base_image kwarg from storage and attempt to deploy a flow, a Cloudpickle error is triggered
Traceback (most recent call last):
  File "/root/.prefect/healthcheck.py", line 12, in <module>
    import cloudpickle
ModuleNotFoundError: No module named 'cloudpickle'




## Expected Behavior
No errors and flow deploys as expected (note: using kwarg base_image="prefecthq/prefect:0.7.0-3.7" allowed me to deploy my flow successfully)



## Reproduction
Remove base_image kwarg from storage and deploy flow




## Environment
Running Core 0.7.0



Discussion:
@cicdw @wagoodman Do you think this could be due to cached layers? I haven't encountered this yet and I often don't provide a base image.
@nanseay could you include what version of Python you are running?
Yep, Python 3.7.3
We looked at this offline and it seems like it's an issue with assumptions on the Docker storage. If you provide that your `base_image` is `python:3.7` and you specify a `prefect_version` of `0.7.0` it won't install that version of prefect (no matter what you provide). This is due to the assumption that if a base image is provided the docker storage does not perform the extra prefect installation.

https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/storage/docker.py#L94
After more discussion, we think the best way forward is to always attempt to install the prefect package unless it is already installed. For example, regardless of the `base_image` or `prefect_version` there is an expectation that prefect will be installed, thus we should always attempt to install it. However, we should check for existing installations (via `pip show prefect`) and do not reinstall if it already exists.

Additionally, it would be good to check if the user specified a `prefect_version` that the final installed version matches what the user provided. This will be a safety measure for when a user brings a specific image that already has prefect installed, and additionally provides a `prefect_version` (which mismatches the version from the base image)... we want the flow deploy to fail since there is an unexpected version in use.

Resolving patch:
diff --git a/src/prefect/environments/storage/docker.py b/src/prefect/environments/storage/docker.py
--- a/src/prefect/environments/storage/docker.py
+++ b/src/prefect/environments/storage/docker.py
@@ -102,17 +102,20 @@ def __init__(
             else:
                 # create an image from python:*-slim directly
                 self.base_image = "python:{}-slim".format(python_version)
-                self.extra_commands.extend(
-                    [
-                        "apt update && apt install -y gcc git && rm -rf /var/lib/apt/lists/*",
-                        "pip install git+https://github.com/PrefectHQ/prefect.git@{}#egg=prefect[kubernetes]".format(
-                            self.prefect_version
-                        ),
-                    ]
+                self.extra_commands.append(
+                    "apt update && apt install -y gcc git && rm -rf /var/lib/apt/lists/*",
                 )
         else:
             self.base_image = base_image
 
+        # we should always try to install prefect, unless it is already installed. We can't determine this until
+        # image build time.
+        self.extra_commands.append(
+            "pip show prefect || pip install git+https://github.com/PrefectHQ/prefect.git@{}#egg=prefect[kubernetes]".format(
+                self.prefect_version
+            ),
+        )
+
         not_absolute = [
             file_path for file_path in self.files if not os.path.isabs(file_path)
         ]


Test patch:
diff --git a/tests/environments/storage/test_docker_storage.py b/tests/environments/storage/test_docker_storage.py
--- a/tests/environments/storage/test_docker_storage.py
+++ b/tests/environments/storage/test_docker_storage.py
@@ -316,7 +316,7 @@ def test_create_dockerfile_from_base_image():
             "master",
             (
                 "FROM python:3.6-slim",
-                "pip install git+https://github.com/PrefectHQ/prefect.git@master",
+                "pip show prefect || pip install git+https://github.com/PrefectHQ/prefect.git@master",
             ),
         ),
         (
@@ -324,7 +324,7 @@ def test_create_dockerfile_from_base_image():
             (
                 "FROM python:3.6-slim",
                 "apt update && apt install -y gcc git && rm -rf /var/lib/apt/lists/*",
-                "pip install git+https://github.com/PrefectHQ/prefect.git@424be6b5ed8d3be85064de4b95b5c3d7cb665510#egg=prefect[kubernetes]",
+                "pip show prefect || pip install git+https://github.com/PrefectHQ/prefect.git@424be6b5ed8d3be85064de4b95b5c3d7cb665510#egg=prefect[kubernetes]",
             ),
         ),
     ],