Repository: canonical/charmcraft

Issue:
spread tests: notebook-operators failing to build
### Bug Description

https://github.com/canonical/charmcraft/actions/runs/5593782587

### To Reproduce

1. `git clone https://github.com/canonical/notebook-operators`
2. `cd notebook-operators`
3. `charmcraft pack --include-all-charms`

This doesn't seem to be an issue on 2.3.1, so it could be fixed by the unmerged fixes at https://github.com/canonical/charmcraft/pull/1165

### Environment

CI

### charmcraft.yaml

```shell
Happens for both the jupyter-controller and jupyter-ui charms: https://github.com/canonical/notebook-operators
```


### Relevant log output

```shell
2023-07-19 14:49:28.659 :: ::    :: Collecting pyyaml==5.4 (from -r requirements.txt (line 56))
2023-07-19 14:49:28.690 :: ::    ::   Downloading PyYAML-5.4.tar.gz (174 kB)
2023-07-19 14:49:28.700 :: ::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.8/174.8 kB 24.3 MB/s eta 0:00:00
2023-07-19 14:49:28.822 :: ::    ::   Installing build dependencies: started
2023-07-19 14:51:25.544 :: ::    ::   Installing build dependencies: still running...
2023-07-19 14:52:33.307 :: ::    ::   Installing build dependencies: still running...
2023-07-19 14:53:33.207 :: ::    ::   Installing build dependencies: finished with status 'done'
2023-07-19 14:53:33.207 :: ::    ::   Getting requirements to build wheel: started
2023-07-19 14:53:33.460 :: ::    ::   Getting requirements to build wheel: finished with status 'error'
2023-07-19 14:53:33.470 :: ::    ::   error: subprocess-exited-with-error
2023-07-19 14:53:33.470 :: ::    ::
2023-07-19 14:53:33.470 :: ::    ::   × Getting requirements to build wheel did not run successfully.
2023-07-19 14:53:33.475 :: ::    ::   │ exit code: 1
2023-07-19 14:53:33.475 :: ::    ::   ╰─> [62 lines of output]
2023-07-19 14:53:33.475 :: ::    ::       running egg_info
2023-07-19 14:53:33.475 :: ::    ::       writing lib3/PyYAML.egg-info/PKG-INFO
2023-07-19 14:53:33.475 :: ::    ::       writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
2023-07-19 14:53:33.475 :: ::    ::       writing top-level names to lib3/PyYAML.egg-info/top_level.txt
2023-07-19 14:53:33.475 :: ::    ::       /tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
2023-07-19 14:53:33.475 :: ::    ::       !!
2023-07-19 14:53:33.478 :: ::    ::
2023-07-19 14:53:33.478 :: ::    ::               ********************************************************************************
2023-07-19 14:53:33.478 :: ::    ::               The license_file parameter is deprecated, use license_files instead.
2023-07-19 14:53:33.478 :: ::    ::
2023-07-19 14:53:33.478 :: ::    ::               By 2023-Oct-30, you need to update your project and remove deprecated calls
2023-07-19 14:53:33.478 :: ::    ::               or your builds will no longer be supported.
2023-07-19 14:53:33.478 :: ::    ::
2023-07-19 14:53:33.478 :: ::    ::               See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
2023-07-19 14:53:33.478 :: ::    ::               ********************************************************************************
2023-07-19 14:53:33.478 :: ::    ::
2023-07-19 14:53:33.478 :: ::    ::       !!
2023-07-19 14:53:33.478 :: ::    ::         parsed = self.parsers.get(option_name, lambda x: x)(value)
2023-07-19 14:53:33.478 :: ::    ::       Traceback (most recent call last):
2023-07-19 14:53:33.478 :: ::    ::         File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
2023-07-19 14:53:33.478 :: ::    ::           main()
2023-07-19 14:53:33.478 :: ::    ::         File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
2023-07-19 14:53:33.478 :: ::    ::           json_out['return_val'] = hook(**hook_input['kwargs'])
2023-07-19 14:53:33.478 :: ::    ::         File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
2023-07-19 14:53:33.478 :: ::    ::           return hook(config_settings)
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
2023-07-19 14:53:33.478 :: ::    ::           return self._get_build_requires(config_settings, requirements=['wheel'])
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
2023-07-19 14:53:33.478 :: ::    ::           self.run_setup()
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in run_setup
2023-07-19 14:53:33.478 :: ::    ::           exec(code, locals())
2023-07-19 14:53:33.478 :: ::    ::         File "<string>", line 271, in <module>
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 107, in setup
2023-07-19 14:53:33.478 :: ::    ::           return distutils.core.setup(**attrs)
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
2023-07-19 14:53:33.478 :: ::    ::           return run_commands(dist)
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
2023-07-19 14:53:33.478 :: ::    ::           dist.run_commands()
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
2023-07-19 14:53:33.478 :: ::    ::           self.run_command(cmd)
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
2023-07-19 14:53:33.478 :: ::    ::           super().run_command(command)
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
2023-07-19 14:53:33.478 :: ::    ::           cmd_obj.run()
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 314, in run
2023-07-19 14:53:33.478 :: ::    ::           self.find_sources()
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
2023-07-19 14:53:33.478 :: ::    ::           mm.run()
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 551, in run
2023-07-19 14:53:33.478 :: ::    ::           self.add_defaults()
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
2023-07-19 14:53:33.478 :: ::    ::           sdist.add_defaults(self)
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
2023-07-19 14:53:33.478 :: ::    ::           super().add_defaults()
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
2023-07-19 14:53:33.478 :: ::    ::           self._add_defaults_ext()
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
2023-07-19 14:53:33.478 :: ::    ::           self.filelist.extend(build_ext.get_source_files())
2023-07-19 14:53:33.478 :: ::    ::         File "<string>", line 201, in get_source_files
2023-07-19 14:53:33.478 :: ::    ::         File "/tmp/pip-build-env-2ei64txa/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
2023-07-19 14:53:33.478 :: ::    ::           raise AttributeError(attr)
2023-07-19 14:53:33.478 :: ::    ::       AttributeError: cython_sources
2023-07-19 14:53:33.478 :: ::    ::       [end of output]
2023-07-19 14:53:33.479 :: ::    ::
2023-07-19 14:53:33.479 :: ::    ::   note: This error originates from a subprocess, and is likely not a problem with pip.
2023-07-19 14:53:33.479 :: ::    :: error: subprocess-exited-with-error
2023-07-19 14:53:33.479 :: ::    ::
2023-07-19 14:53:33.479 :: ::    :: × Getting requirements to build wheel did not run successfully.
2023-07-19 14:53:33.479 :: ::    :: │ exit code: 1
2023-07-19 14:53:33.479 :: ::    :: ╰─> See above for output.
2023-07-19 14:53:33.479 :: ::    ::
2023-07-19 14:53:33.479 :: ::    :: note: This error originates from a subprocess, and is likely not a problem with pip.
2023-07-19 14:53:33.826 :: :: Traceback (most recent call last):
2023-07-19 14:53:33.833 :: ::   File "/snap/charmcraft/1434/lib/charmcraft/charm_builder.py", line 421, in <module>
2023-07-19 14:53:33.847 :: ::     main()
2023-07-19 14:53:33.847 :: ::   File "/snap/charmcraft/1434/lib/charmcraft/charm_builder.py", line 416, in main
2023-07-19 14:53:33.847 :: ::     builder.build_charm()
2023-07-19 14:53:33.847 :: ::   File "/snap/charmcraft/1434/lib/charmcraft/charm_builder.py", line 89, in build_charm
2023-07-19 14:53:33.847 :: ::     self.handle_dependencies()
2023-07-19 14:53:33.847 :: ::   File "/snap/charmcraft/1434/lib/charmcraft/charm_builder.py", line 303, in handle_dependencies
2023-07-19 14:53:33.848 :: ::     self._install_dependencies(staging_venv_dir)
2023-07-19 14:53:33.848 :: ::   File "/snap/charmcraft/1434/lib/charmcraft/instrum.py", line 152, in _f
2023-07-19 14:53:33.848 :: ::     return func(*args, **kwargs)
2023-07-19 14:53:33.848 :: ::   File "/snap/charmcraft/1434/lib/charmcraft/charm_builder.py", line 257, in _install_dependencies
2023-07-19 14:53:33.849 :: ::     _process_run(cmd)
2023-07-19 14:53:33.849 :: ::   File "/snap/charmcraft/1434/lib/charmcraft/charm_builder.py", line 357, in _process_run
2023-07-19 14:53:33.849 :: ::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
2023-07-19 14:53:33.849 :: :: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip3', 'install', '--upgrade', '--no-binary', ':all:', '--requirement=requirements.txt'] execution failed with retcode 1
2023-07-19 14:53:34.152 :: Parts processing error: Failed to run the build script for part 'charm'.
2023-07-19 14:53:37.235 :: Failed to build charm for bases index '0'.
2023-07-19 14:53:37.251 :: Full execution log: '/home/lengau/.local/state/charmcraft/log/charmcraft-20230719-144743.734332.log'
2023-07-19 14:53:37.785 charmcraft internal error: CalledProcessError(1, ['charmcraft', 'pack', '--verbose', '--project-dir=/home/lengau/Work/Test/charms/notebook-operators/charms/jupyter-ui'])
```


Discussion:

Patch:
diff --git a/charmcraft/charm_builder.py b/charmcraft/charm_builder.py
index 7189c795..213e2f94 100644
--- a/charmcraft/charm_builder.py
+++ b/charmcraft/charm_builder.py
@@ -25,8 +25,8 @@ import hashlib
 import os
 import pathlib
 import shutil
-import sys
 import subprocess
+import sys
 from typing import List
 
 from charmcraft import instrum
@@ -62,7 +62,7 @@ class CharmBuilder:
         binary_python_packages: List[str] = None,
         python_packages: List[str] = None,
         requirements: List[pathlib.Path] = None,
-    ):
+    ) -> None:
         self.builddir = builddir
         self.installdir = installdir
         self.entrypoint = entrypoint
@@ -249,17 +249,15 @@ class CharmBuilder:
                 cmd.extend(self.python_packages)  # the python packages to install
                 _process_run(cmd)
 
-            if self.requirement_paths:
-                # install dependencies from requirement files
+            if self.requirement_paths or self.charmlib_deps:
                 cmd = [pip_cmd, "install", "--upgrade", "--no-binary", ":all:"]  # base command
-                for reqspath in self.requirement_paths:
-                    cmd.append("--requirement={}".format(reqspath))  # the dependencies file(s)
-                _process_run(cmd)
 
-            if self.charmlib_deps:
-                # install charmlibs python dependencies
-                cmd = [pip_cmd, "install", "--upgrade", "--no-binary", ":all:"]  # base command
-                cmd.extend(self.charmlib_deps)  # the python packages to install
+                if self.requirement_paths:
+                    # install dependencies from requirement files
+                    cmd.extend(f"--requirement={path}" for path in self.requirement_paths)
+
+                if self.charmlib_deps:
+                    cmd.extend(self.charmlib_deps)
                 _process_run(cmd)
 
     def handle_dependencies(self):


Test patch:
diff --git a/tests/spread/smoketests/pinned-dependencies/task.yaml b/tests/spread/smoketests/pinned-dependencies/task.yaml
new file mode 100644
index 00000000..a38baf7d
--- /dev/null
+++ b/tests/spread/smoketests/pinned-dependencies/task.yaml
@@ -0,0 +1,32 @@
+summary: pack a charm with a specific (old) dependency version
+
+include:
+  - tests/
+
+prepare: |
+  tests.pkgs install unzip
+  charmcraft init --project-dir=charm
+  cd charm
+  echo "ops==2.0.0" > requirements.txt
+  cat <<- EOF >> charmcraft.yaml
+  parts:
+    charm:
+      charm-requirements: [requirements.txt]
+  EOF
+  mkdir -p lib/charms/charm/v0/
+  cat <<- EOF > lib/charms/charm/v0/my_lib.py
+  PYDEPS = ["ops"]
+  LIBID = "my_lib"
+  LIBAPI = 0
+  LIBPATCH = 1
+  EOF
+restore: |
+  pushd charm
+  charmcraft clean
+  popd
+  rm -rf charm
+execute: |
+  cd charm
+  charmcraft pack --verbose
+  test -f charm*.charm
+  unzip -p charm_*.charm venv/ops/version.py | MATCH "version = '2.0.0'"
diff --git a/tests/test_charm_builder.py b/tests/test_charm_builder.py
index fa574996..692121fa 100644
--- a/tests/test_charm_builder.py
+++ b/tests/test_charm_builder.py
@@ -791,9 +791,10 @@ def test_build_dependencies_virtualenv_all(tmp_path, assert_output):
                 ":all:",
                 f"--requirement={reqs_file_1}",
                 f"--requirement={reqs_file_2}",
+                "pkg5",
+                "pkg6",
             ]
         ),
-        call([pip_cmd, "install", "--upgrade", "--no-binary", ":all:", "pkg5", "pkg6"]),
     ]
 
     site_packages_dir = charm_builder._find_venv_site_packages(pathlib.Path(STAGING_VENV_DIRNAME))
diff --git a/tests/test_providers.py b/tests/test_providers.py
index feec580f..8913887a 100644
--- a/tests/test_providers.py
+++ b/tests/test_providers.py
@@ -18,7 +18,7 @@ import contextlib
 import pathlib
 import re
 import sys
-from unittest.mock import Mock, patch, call
+from unittest.mock import Mock, call, patch
 
 import pytest
 from craft_cli import CraftError