Repository: spack/spack

Issue:
Installation issue: python+ctypes
### Steps to reproduce the issue

```console
$ spack spec -I python
Input spec
--------------------------------
 -   python

Concretized
--------------------------------
[+]  python@3.9.10%gcc@8.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-centos7-x86_64
[+]      ^bzip2@1.0.8%gcc@8.5.0~debug~pic+shared arch=linux-centos7-x86_64
[+]          ^diffutils@3.8%gcc@8.5.0 arch=linux-centos7-x86_64
[+]              ^libiconv@1.16%gcc@8.5.0 libs=shared,static arch=linux-centos7-x86_64
[+]      ^expat@2.4.6%gcc@8.5.0+libbsd arch=linux-centos7-x86_64
[+]          ^libbsd@0.11.5%gcc@8.5.0 arch=linux-centos7-x86_64
[+]              ^libmd@1.0.4%gcc@8.5.0 arch=linux-centos7-x86_64
[+]      ^gdbm@1.19%gcc@8.5.0 arch=linux-centos7-x86_64
[+]          ^readline@8.1%gcc@8.5.0 arch=linux-centos7-x86_64
[+]              ^ncurses@6.2%gcc@8.5.0~symlinks+termlib abi=none arch=linux-centos7-x86_64
[+]                  ^pkgconf@1.8.0%gcc@8.5.0 arch=linux-centos7-x86_64
[+]      ^gettext@0.21%gcc@8.5.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos7-x86_64
[+]          ^libxml2@2.9.12%gcc@8.5.0~python arch=linux-centos7-x86_64
[+]              ^xz@5.2.5%gcc@8.5.0~pic libs=shared,static arch=linux-centos7-x86_64
[+]              ^zlib@1.2.11%gcc@8.5.0+optimize+pic+shared arch=linux-centos7-x86_64
[+]          ^tar@1.34%gcc@8.5.0 arch=linux-centos7-x86_64
[+]      ^libffi@3.4.2%gcc@8.5.0 arch=linux-centos7-x86_64
[+]      ^openssl@1.1.1m%gcc@8.5.0~docs certs=system arch=linux-centos7-x86_64
[+]          ^perl@5.34.0%gcc@8.5.0+cpanm+shared+threads arch=linux-centos7-x86_64
[+]              ^berkeley-db@18.1.40%gcc@8.5.0+cxx~docs+stl patches=b231fcc arch=linux-centos7-x86_64
[+]      ^sqlite@3.37.2%gcc@8.5.0+column_metadata+dynamic_extensions+fts~functions+rtree arch=linux-centos7-x86_64
[+]      ^util-linux-uuid@2.37.4%gcc@8.5.0 arch=linux-centos7-x86_64
```


### Error message

Python appears to install successfully, but downstream packages fail, e.g. py-setuptools:
<details><summary>Error message</summary><pre>
```
==> py-setuptools: Executing phase: 'install'
==> [2022-03-12-10:39:09.181041] '/projects/spack/opt/spack/gcc-8.5.0/python/6z6bbfg/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/projects/spack/opt/spack/gcc-8.5.0/py-setuptools/26gyn5x' '.'
Using pip 21.3.1 from /projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip (python 3.9)
Non-user install due to --prefix or --target option
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-p5vr3e0t
Created temporary directory: /tmp/pip-req-tracker-iv7wygbd
Initialized build tracking at /tmp/pip-req-tracker-iv7wygbd
Created build tracker: /tmp/pip-req-tracker-iv7wygbd
Entered build tracker: /tmp/pip-req-tracker-iv7wygbd
Created temporary directory: /tmp/pip-install-2zg9v9l0
Processing /tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src
  Added file:///tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src to build tracker '/tmp/pip-req-tracker-iv7wygbd'
  Created temporary directory: /tmp/pip-modern-metadata-n1k48pa1
  Preparing metadata (pyproject.toml): started
  Running command /projects/spack/opt/spack/gcc-8.5.0/python/6z6bbfg/bin/python3.9 /projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpte_wdk3q
  Preparing metadata (pyproject.toml): finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 338, in run
    requirement_set = resolver.resolve(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 430, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
    super().__init__(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 546, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 49, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 556, in prepare_metadata
    self.metadata_directory = generate_metadata(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/operations/build/metadata.py", line 28, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 188, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
    obj = import_module(mod_path)
  File "/projects/spack/opt/spack/gcc-8.5.0/python/6z6bbfg/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src/setuptools/__init__.py", line 18, in <module>
    from setuptools.dist import Distribution
  File "/tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src/setuptools/dist.py", line 38, in <module>
    from setuptools import windows_support
  File "/tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/projects/spack/opt/spack/gcc-8.5.0/python/6z6bbfg/lib/python3.9/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
</pre></details>

Looking back at the python build logs, it silently hid the failure of the ctypes module:
<details><summary>Error message</summary><pre>
*** WARNING: renaming "_ctypes" since importing it failed: build/lib.linux-x86_64-3.9/_ctypes.cpython-39-x86_64-linux-gnu.so: undefined symbol: ffi_prep_cif

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_lzma                 _sqlite3              _tkinter
readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Following modules built successfully but were removed because they could not be imported:
_ctypes
</pre></details>

### Information on your system

* **Spack:** 0.17.1-1532-8adc6b7e8e
* **Python:** 3.9.10
* **Platform:** linux-centos7-zen2
* **Concretizer:** clingo

### Additional information

@adamjstewart @scheibelp @skosukhin @varioustoxins

A recent build on an almost identical configuration passed, so I suspect this is due to the libffi update in #28970 .
[spack-build-out.txt](https://github.com/spack/spack/files/8238107/spack-build-out.txt)

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
Update: this might somehow be a change in the core spack build or some other environmental issue. I reinstalled a known-working build (same hash!) of python+ctypes (using the older libffi) and whereas the working build adds `-lffi` to the linker args for `_ctypes`, the rebuild does not! 😱 

Dammit, looks like the configure line changed so that it's no longer adding CPPFLAGS/LDFLAGS:
```diff
 ==> python: Executing phase: 'configure'
-==> [2022-02-10-17:13:47.632468] '/tmp/s3j/spack-stage/spack-stage-python-3.9.10-6h5h57afws4qoyvloh3cb4bhyyxaggl7/spack-src/configure' '--prefix=/projects/spack/opt/spack/gcc-8.5.0/python/6h5h57a' 'CPPFLAGS=-I/projects/spack/opt/spack/gcc-8.5.0/bzip2/pwzu7zw/include -I/projects/spack/opt/spack/gcc-8.5.0/expat/skpazwg/include -I/projects/spack/opt/spack/gcc-8.5.0/gdbm/67yjsyn/include -I/projects/spack/opt/spack/gcc-8.5.0/gettext/h27562m/include -I/projects/spack/opt/spack/gcc-8.5.0/libffi/wplppns/include -I/projects/spack/opt/spack/gcc-8.5.0/ncurses/ppry2fz/include -I/projects/spack/opt/spack/gcc-8.5.0/openssl/qqgwhbn/include -I/projects/spack/opt/spack/gcc-8.5.0/readline/nph3lge/include -I/projects/spack/opt/spack/gcc-8.5.0/sqlite/ggm6u65/include -I/projects/spack/opt/spack/gcc-8.5.0/util-linux-uuid/yhkg22d/include -I/projects/spack/opt/spack/gcc-8.5.0/xz/6gols7g/include -I/projects/spack/opt/spack/gcc-8.5.0/zlib/draacp6/include' 'LDFLAGS=-L/projects/spack/opt/spack/gcc-8.5.0/bzip2/pwzu7zw/lib -L/projects/spack/opt/spack/gcc-8.5.0/expat/skpazwg/lib -L/projects/spack/opt/spack/gcc-8.5.0/gdbm/67yjsyn/lib -L/projects/spack/opt/spack/gcc-8.5.0/gettext/h27562m/lib -L/projects/spack/opt/spack/gcc-8.5.0/libffi/wplppns/lib64 -L/projects/spack/opt/spack/gcc-8.5.0/ncurses/ppry2fz/lib -L/projects/spack/opt/spack/gcc-8.5.0/openssl/qqgwhbn/lib -L/projects/spack/opt/spack/gcc-8.5.0/readline/nph3lge/lib -L/projects/spack/opt/spack/gcc-8.5.0/sqlite/ggm6u65/lib -L/projects/spack/opt/spack/gcc-8.5.0/util-linux-uuid/yhkg22d/lib -L/projects/spack/opt/spack/gcc-8.5.0/xz/6gols7g/lib -L/projects/spack/opt/spack/gcc-8.5.0/zlib/draacp6/lib' '--without-pydebug' '--enable-shared' '--with-ensurepip=install' '--with-openssl=/projects/spack/opt/spack/gcc-8.5.0/openssl/qqgwhbn' '--with-dbmliborder=gdbm:bdb:ndbm' '--with-system-expat' '--with-system-ffi' '--enable-loadable-sqlite-extensions' 'CFLAGS=-fPIC'
+==> [2022-03-12-13:46:34.814390] '/tmp/s3j/spack-stage/spack-stage-python-3.9.10-6h5h57afws4qoyvloh3cb4bhyyxaggl7/spack-src/configure' '--prefix=/projects/spack/opt/spack/gcc-8.5.0/python/6h5h57a' '--without-pydebug' '--enable-shared' '--with-ensurepip=install' '--with-openssl=/projects/spack/opt/spack/gcc-8.5.0/openssl/qqgwhbn' '--with-dbmliborder=gdbm:bdb:ndbm' '--with-system-expat' '--with-system-ffi' '--enable-loadable-sqlite-extensions' 'CFLAGS=-fPIC'
 checking build system type... x86_64-pc-linux-gnu
```
OK confirmed, reverting #28673 reinstates the LDFLAGS/CPPFLAGS and `_ctypes` builds again.
I can also confirm that reverting #28673 fixes the build failure.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -443,7 +443,7 @@ def configure_args(self):
 
         # setup.py needs to be able to read the CPPFLAGS and LDFLAGS
         # as it scans for the library and headers to build
-        link_deps = spec.dependencies('link')
+        link_deps = spec.dependencies(deptype='link')
 
         if link_deps:
             # Header files are often included assuming they reside in a


Test patch: