Repository: nautobot/nautobot

Issue:
Different dependency versions installed in `dev` versus `final` Docker images
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version: any
* Nautobot version: 1.1.1

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Build and run `dev` docker image and run `pip list`
2. Build and run `final` docker image and run `pip list`
3. Compare outputs

<!-- What did you expect to happen? -->
### Expected Behavior

Installed packages should be identical except for the additional dev-only dependencies present in the `dev` image.

<!-- What happened instead? -->
### Observed Behavior

In several cases `final` has newer versions of various dependencies than `dev`, for example:

- `dev` has `idna 2.10` whereas `final` has `idna 3.2`
- `dev` has `packaging 20.9` whereas `final` has `packaging 21.0`
- `dev` has `requests 2.25.1` whereas `final` has `requests 2.26.0`
- etc.

This is problematic as it means that `final` and `dev` may show different behavior, have different bugs or fixes, etc.

The root cause appears to be that the `dev` image installs its dependencies based on exact versions pinned in `poetry.lock`, whereas the `final` image installs its dependencies based on the looser version constraints defined in `pyproject.toml`.

Related Poetry FR and PR: https://github.com/python-poetry/poetry/issues/2778, https://github.com/python-poetry/poetry/pull/3341
The canonical source for Poetry's installation script is now https://install.python-poetry.org. Please update your usage to reflect this.
### Proposed Changes

As seen during Docker build:

```
#9 [dev  3/14] RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py -o /tmp/install-poetry.py &&     python /tmp/install-poetry.py &&     rm -f /tmp/install-poetry.py
#9 sha256:4698fc6f679394c044d5fac676800af0f89d000fa6891658584f3153722db27b
#9 0.808 The canonical source for Poetry's installation script is now https://install.python-poetry.org. Please update your usage to reflect this.
```


Discussion:
One possibility might be to use https://bneijt.nl/pr/poetry-lock-package/ to build a "locked" wheel and install that one in the `final` Docker image so as to ensure that it has the same dependencies as the `dev` image. The potential downside is that this will prevent any consumers of the `final` image from doing a `pip install --upgrade` of any dependencies, even for security updates and such, since the dependency versions will be locked.

Another option would be to aggressively ensure that for our release version images, at least, that we are consistently running `poetry update` prior to release so as to minimize the delta between what's in `dev` images and what are the latest acceptable versions of our dependencies installed in `final` images.
Related: #1097.
Reopening as #1323 had to be rolled back by #1325. 
There seem to be quite a few open issues against Poetry involving incorrect dependency declarations - any or all of these may be causing the issues we encountered with #1323, but dropping support for Python 3.6 (#1268) will at least in the short term allow us to avoid the currently encountered issue as it'll let us remove the split dependencies that we currently have for Python 3.6 versus all later versions, which are causing the current failures. However there is a concern that at some point in the future we may find ourselves in this same position (e.g. with respect to Python 3.7) so getting the issues fixed in Poetry would still be highly desirable.

- https://github.com/python-poetry/poetry/issues/4428
- https://github.com/python-poetry/poetry/issues/4461
- https://github.com/python-poetry/poetry/issues/4810
- https://github.com/python-poetry/poetry/issues/4209
- https://github.com/python-poetry/poetry/issues/3959
- https://github.com/python-poetry/poetry-export-plugin/issues/25
- https://github.com/python-poetry/poetry-export-plugin/issues/32


Resolving patch:
diff --git a/tasks.py b/tasks.py
--- a/tasks.py
+++ b/tasks.py
@@ -136,7 +136,11 @@ def run_command(context, command, **kwargs):
 )
 def build(context, force_rm=False, cache=True):
     """Build Nautobot docker image."""
-    command = f"build --build-arg PYTHON_VER={context.nautobot.python_ver}"
+    command = (
+        "build"
+        f" --build-arg PYTHON_VER={context.nautobot.python_ver}"
+        f" --build-arg PYUWSGI_VER={get_dependency_version('pyuwsgi')}"
+    )
 
     if not cache:
         command += " --no-cache"
@@ -166,11 +170,19 @@ def buildx(
 ):
     """Build Nautobot docker image using the experimental buildx docker functionality (multi-arch capablility)."""
     print(f"Building Nautobot with Python {context.nautobot.python_ver} for {platforms}...")
-    command = f"docker buildx build --platform {platforms} -t {tag} --target {target} --load -f ./docker/Dockerfile --build-arg PYTHON_VER={context.nautobot.python_ver} ."
+    command = (
+        f"docker buildx build --platform {platforms} -t {tag} --target {target} --load -f ./docker/Dockerfile"
+        f" --build-arg PYTHON_VER={context.nautobot.python_ver}"
+        f" --build-arg PYUWSGI_VER={get_dependency_version('pyuwsgi')}"
+        " ."
+    )
     if not cache:
         command += " --no-cache"
     else:
-        command += f" --cache-to type=local,dest={cache_dir}/{context.nautobot.python_ver} --cache-from type=local,src={cache_dir}/{context.nautobot.python_ver}"
+        command += (
+            f" --cache-to type=local,dest={cache_dir}/{context.nautobot.python_ver}"
+            f" --cache-from type=local,src={cache_dir}/{context.nautobot.python_ver}"
+        )
 
     context.run(command, env={"PYTHON_VER": context.nautobot.python_ver})
 
@@ -184,6 +196,15 @@ def get_nautobot_version():
     return version_match[0]
 
 
+def get_dependency_version(dependency_name):
+    """Get the version of a given direct dependency from `pyproject.toml`."""
+    with open("pyproject.toml", "r") as fh:
+        content = fh.read()
+
+    version_match = re.search(rf'^{dependency_name} = .*"[~^]?([0-9.]+)"', content, flags=re.MULTILINE)
+    return version_match.group(1)
+
+
 @task(
     help={
         "branch": "Source branch used to push.",


Test patch: