Repository: spack/spack

Issue:
Installation issue: py-pkgconfig needs py-poetry, not py-poetry-core?
### Steps to reproduce the issue

Trying to install `py-h5py` which needs `py-pkgconfig`. The installation of `py-pkgconfig` fails with
```console
$  spack install -v py-pkgconfig
...
  File "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'poetry'
```
Package `py-poetry-core` is installed, which is what these packages want, according to their dependency list. Trying to install `py-poetry` to fix this leads to the same error for one of the `py-poetry` dependencies, `py-crashtest`:
```
...
  File "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'poetry'
```
So, you can't install `py-poetry` because it needs `py-poetry` installed it seems.

There is some limited information here, where pipenv fails with the same error (at least for their release version, but fixed in master): https://github.com/pypa/pipenv/issues/4218

Any suggestions on how to fix that? Was that introduced with https://github.com/spack/spack/pull/27798? Are there other upcoming developments that I could pull in? I don't really want to install poetry using homebrew, because this installs a lot more into the otherwise clean python3 installation. Note: my python was installed via homebrew, but I didn't want to add any packages there, do the rest with spack.

### Error message

```
  File "/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'poetry'
```


### Information on your system

```
* **Spack:** 0.17.1-1147-15f83b6e93
* **Python:** 3.9.10
* **Platform:** darwin-monterey-x86_64
* **Concretizer:** clingo
```

Python 3.10 installed via homebrew, all other packages in a spack env.

### Additional information

[spack-build-env.txt](https://github.com/spack/spack/files/8173693/spack-build-env.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/8173694/spack-build-out.txt)

No maintainers for any of these packages.

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
This posts suggests to use `pipx` to install poetry: https://github.com/python-poetry/poetry/issues/2449
Ok, my workaround is to use homebrew's Python's pip3 to install poetry and then the rest just falls in place. Not the ideal solution, but it works.
I'm unable to reproduce this, both `py-pkgconfig` and `py-crashtest` build fine for me on develop. The full build and logs would be helpful in understanding why this is happening.
I'll try to retrieve those. It just happened again to me on a completely different system (HPC - MSU Orion), and I solved it by ditching the Python module that the sysadmins provide and installing a miniconda3 version that has `poetry` installed via pip.
I got this error on several more systems in the last two weeks. We worked around it by installing `poetry` (not just the core part) outside of spack for each of the Python installations. I still think that this is a problem, but I don't have the time to investigate this much further right now.
@adamjstewart I ran into this again. Here is the full output when trying to install pkgconfig with a Python that was installed via spack:
```
==> Installing py-pkgconfig-1.5.5-hpunqjarrjn4g7sziv7dulisuu3n7l2f
==> No binary for py-pkgconfig-1.5.5-hpunqjarrjn4g7sziv7dulisuu3n7l2f found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/de/deb4163ef11f75b520d822d9505c1f462761b4309b1bb713d08689759ea8b899.tar.gz
==> No patches needed for py-pkgconfig
==> py-pkgconfig: Executing phase: 'install'
==> [2022-03-28-22:27:24.706937] '/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.8.9-l6r5ldx/bin/python3.8' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pkgconfig-1.5.5-hpunqja' '.'
Using pip 21.3.1 from /home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip (python 3.8)
Non-user install due to --prefix or --target option
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-dpzm3jzi
Created temporary directory: /tmp/pip-req-tracker-ktjy_i7r
Initialized build tracking at /tmp/pip-req-tracker-ktjy_i7r
Created build tracker: /tmp/pip-req-tracker-ktjy_i7r
Entered build tracker: /tmp/pip-req-tracker-ktjy_i7r
Created temporary directory: /tmp/pip-install-w_zpgfeg
Processing /home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-hpunqjarrjn4g7sziv7dulisuu3n7l2f/spack-src
  Added file:///home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-hpunqjarrjn4g7sziv7dulisuu3n7l2f/spack-src to build tracker '/tmp/pip-req-tracker-ktjy_i7r'
  Created temporary directory: /tmp/pip-modern-metadata-iswzz8ge
  Preparing metadata (pyproject.toml): started
  Running command /home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.8.9-l6r5ldx/bin/python3.8 /home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp6egn2oz8
  Preparing metadata (pyproject.toml): finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 338, in run
    requirement_set = resolver.resolve(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 430, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
    super().__init__(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 546, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 49, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 556, in prepare_metadata
    self.metadata_directory = generate_metadata(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_internal/operations/build/metadata.py", line 28, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 188, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-asgl4kf/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
    obj = import_module(mod_path)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.8.9-l6r5ldx/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'poetry'

Removed file:///home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-hpunqjarrjn4g7sziv7dulisuu3n7l2f/spack-src from build tracker '/tmp/pip-req-tracker-ktjy_i7r'
Removed build tracker: '/tmp/pip-req-tracker-ktjy_i7r'
==> Error: ProcessError: Command exited with status 2:
    '/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.8.9-l6r5ldx/bin/python3.8' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pkgconfig-1.5.5-hpunqja' '.'
See build log for details:
  /home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-hpunqjarrjn4g7sziv7dulisuu3n7l2f/spack-build-out.txt

==> Warning: Skipping build of py-h5py-3.6.0-bfv2anah3wuqfk3yso7yulgduhtyorz5 since py-pkgconfig-1.5.5-hpunqjarrjn4g7sziv7dulisuu3n7l2f failed
==> Warning: Skipping build of jedi-base-env-main-3263hiql76vtjzmy5krrrpwpedo3x25a since py-h5py-3.6.0-bfv2anah3wuqfk3yso7yulgduhtyorz5 failed
==> Warning: Skipping build of soca-bundle-env-main-3zno4gkyz6h2elfcn6x3qxxeiwxfp36r since jedi-base-env-main-3263hiql76vtjzmy5krrrpwpedo3x25a failed
==> Warning: Skipping build of jedi-um-bundle-env-main-y7wufvvkh6p4e3bezq6yxbr6uydsdvh6 since jedi-base-env-main-3263hiql76vtjzmy5krrrpwpedo3x25a failed
==> Warning: Skipping build of jedi-ufs-bundle-env-main-xrdjlqel6pfqxrtit4x2ivwojmfpgjfg since jedi-base-env-main-3263hiql76vtjzmy5krrrpwpedo3x25a failed
==> Warning: Skipping build of jedi-fv3-bundle-env-main-h3dejk6dt3z2uanhsbfaveryaepjn27b since jedi-base-env-main-3263hiql76vtjzmy5krrrpwpedo3x25a failed
==> Error: Terminating after first install failure: ProcessError: Command exited with status 2:
    '/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.8.9-l6r5ldx/bin/python3.8' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pkgconfig-1.5.5-hpunqja' '.'
```
This content is identical to what is in file ` /home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-hpunqjarrjn4g7sziv7dulisuu3n7l2f/spack-build-out.txt`.
@adamjstewart FYI, I also got the problem when I used a Python 3.9 version that was installed via yum (both tests were on a Redhat8 linux box, but I've seen this on many other systems, including MSU Orion, RDHPC Hera, ...
```
==> Installing py-pkgconfig-1.5.5-zxigsswuufp5lm3fyqzyoylxlvhvcfc6
==> No binary for py-pkgconfig-1.5.5-zxigsswuufp5lm3fyqzyoylxlvhvcfc6 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/de/deb4163ef11f75b520d822d9505c1f462761b4309b1bb713d08689759ea8b899.tar.gz
==> No patches needed for py-pkgconfig
==> py-pkgconfig: Executing phase: 'install'
==> [2022-03-28-23:07:28.065219] '/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.9.10-ner3cuu/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pkgconfig-1.5.5-zxigssw' '.'
Using pip 21.3.1 from /home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip (python 3.9)
Non-user install due to --prefix or --target option
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-cx6cjsje
Created temporary directory: /tmp/pip-req-tracker-pyr99uq2
Initialized build tracking at /tmp/pip-req-tracker-pyr99uq2
Created build tracker: /tmp/pip-req-tracker-pyr99uq2
Entered build tracker: /tmp/pip-req-tracker-pyr99uq2
Created temporary directory: /tmp/pip-install-jh86sn9x
Processing /home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-zxigsswuufp5lm3fyqzyoylxlvhvcfc6/spack-src
  Added file:///home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-zxigsswuufp5lm3fyqzyoylxlvhvcfc6/spack-src to build tracker '/tmp/pip-req-tracker-pyr99uq2'
  Created temporary directory: /tmp/pip-modern-metadata-rk0fenle
  Preparing metadata (pyproject.toml): started
  Running command /home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.9.10-ner3cuu/bin/python3.9 /home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpj27prqgi
  Preparing metadata (pyproject.toml): finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 338, in run
    requirement_set = resolver.resolve(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 430, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
    super().__init__(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 546, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 49, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 556, in prepare_metadata
    self.metadata_directory = generate_metadata(
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_internal/operations/build/metadata.py", line 28, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 188, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pip-21.3.1-7fwzvu4/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
    obj = import_module(mod_path)
  File "/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.9.10-ner3cuu/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'poetry'

Removed file:///home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-zxigsswuufp5lm3fyqzyoylxlvhvcfc6/spack-src from build tracker '/tmp/pip-req-tracker-pyr99uq2'
Removed build tracker: '/tmp/pip-req-tracker-pyr99uq2'
==> Error: ProcessError: Command exited with status 2:
    '/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.9.10-ner3cuu/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pkgconfig-1.5.5-zxigssw' '.'
See build log for details:
  /home/ec2-user/sandpit/spack-stack/cache/build_stage/spack-stage-py-pkgconfig-1.5.5-zxigsswuufp5lm3fyqzyoylxlvhvcfc6/spack-build-out.txt

==> Warning: Skipping build of py-h5py-3.6.0-esn5mbvwq2koxvdh5speu7ijzju3gmd7 since py-pkgconfig-1.5.5-zxigsswuufp5lm3fyqzyoylxlvhvcfc6 failed
==> Warning: Skipping build of jedi-base-env-main-7cm43yki5fid5k5xe5mkeq4azqq7juhg since py-h5py-3.6.0-esn5mbvwq2koxvdh5speu7ijzju3gmd7 failed
==> Warning: Skipping build of jedi-um-bundle-env-main-lebttaiisjsv4urpdses3pyvy642pfr7 since jedi-base-env-main-7cm43yki5fid5k5xe5mkeq4azqq7juhg failed
==> Warning: Skipping build of jedi-ufs-bundle-env-main-nnvplc4tdqqxu32bby7747rlrosa5thm since jedi-base-env-main-7cm43yki5fid5k5xe5mkeq4azqq7juhg failed
==> Warning: Skipping build of jedi-fv3-bundle-env-main-t74iuyrf5htxi26jvipkptll5bhkwqii since jedi-base-env-main-7cm43yki5fid5k5xe5mkeq4azqq7juhg failed
==> Warning: Skipping build of soca-bundle-env-main-q36tb27r4bfejf7kgoebquv5xr4bwezq since jedi-base-env-main-7cm43yki5fid5k5xe5mkeq4azqq7juhg failed
==> Error: Terminating after first install failure: ProcessError: Command exited with status 2:
    '/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/python-3.9.10-ner3cuu/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/ec2-user/sandpit/spack-stack/envs/jedi-all-gcc11/install/gcc/11.2.1/py-pkgconfig-1.5.5-zxigssw' '.'
```

I'm still unable to reproduce this on any system I have access to. Can you provide a Dockerfile to reproduce this?
I ran in this again on a RedHat8 AWS EC2 instance. I could make that available to you, if that helps.
Yes, if you could give me access to that instance that would help reproduce this.
@adamjstewart I finally got this ready for you to try. I will send you an email.
@adamjstewart Could reproduce the problem with theAWS EC2 instance I provided. The installation worked fine if Python is built by spack, not used as an external dependency (in this RedHat8 case installed via `yum`).
Probably related to https://github.com/spack/spack/issues/32375, https://github.com/spack/spack/pull/32936, https://github.com/python-poetry/poetry/issues/5547.

TL;DR: poetry-core will use global `.gitignore` files to skip files that get installed, resulting in an empty installation prefix. Current workaround is to remove `poetry` from your global `.gitignore`. I'll see if I can find a more robust solution.
To add to this, I think it's grabbing spack's `.gitignore` and causing issues. I was running into this bug again, and I was able to get around it with a dirty hack:

```
cd ~/Documents/GitHub/spack
mv .gitignore .gitignore.BACKUP
```

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-poetry-core/package.py b/var/spack/repos/builtin/packages/py-poetry-core/package.py
--- a/var/spack/repos/builtin/packages/py-poetry-core/package.py
+++ b/var/spack/repos/builtin/packages/py-poetry-core/package.py
@@ -27,3 +27,10 @@ class PyPoetryCore(PythonPackage):
     depends_on("py-typing@3.7.4.1:3", when="^python@2.7", type=("build", "run"))
     depends_on("py-enum34@1.1.10:1", when="^python@2.7", type=("build", "run"))
     depends_on("py-functools32@3.2.3-2:3", when="^python@2.7", type=("build", "run"))
+
+    # https://github.com/python-poetry/poetry/issues/5547
+    def setup_build_environment(self, env):
+        env.set("GIT_DIR", self.stage.source_path)
+
+    def setup_dependent_build_environment(self, env, dependent_spec):
+        env.set("GIT_DIR", dependent_spec.package.stage.source_path)


Test patch: