Repository: spack/spack

Issue:
clingo bootstrap fails on OSX M1 Max + Xcode (0.19.0dev0)
### Steps to reproduce

Similar to #30958 but not exactly so.
@bhatele Any luck with yours?
```
cd
git clone git@github.com:rountree/spack.git
export SPACK_USER_CONFIG_PATH=${HOME}/spack/user_config
export SPACK_SYSTEM_CONFIG_PATH=${HOME}/spack/system_config
export SPACK_USER_CACHE_PATH=${HOME}/spack/user_cache
export tempdir=${HOME}/spack/tempdir
source ./spack/share/spack/setup-env.sh

spack spec pigz
```

### Error message

```
==> Installing clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj
==> No binary for clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj found: installing from source
==> Error: NameError: name 'directory' is not defined

/Users/rountree4/spack/var/spack/repos/builtin/packages/python/package.py:1115, in headers:
       1112                    break
       1113            else:
       1114                msg = "Unable to locate {} headers in {}"
  >>   1115                raise spack.error.NoHeadersError(msg.format(self.name, directory))
       1116
       1117        headers.directories = [os.path.dirname(config_h)]
       1118        return headers


==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %apple-clang target=aarch64" due to the following failures:
github-actions-v0.1 raised ValueError: source is not trusted
spack-install raised InstallError: Terminating after first install failure: NameError: name 'directory' is not defined
Run `spack --debug ...` for more detailed errors
```


Under `--debug`

```
==> [2022-08-02-21:31:01.724534] Installing clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj
==> [2022-08-02-21:31:01.724555] Searching for binary cache of clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj
==> [2022-08-02-21:31:02.591653] Did not find darwin-monterey-aarch64-apple-clang-13.1.6-clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj.spec.json on https://mirror.spack.io/build_cache/darwin-monterey-aarch64-apple-clang-13.1.6-clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj.spec.json.sig
  Download failed: HTTP Error 404: Not Found
==> [2022-08-02-21:31:02.592055] Did not find darwin-monterey-aarch64-apple-clang-13.1.6-clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj.spec.json on https://mirror.spack.io/build_cache/darwin-monterey-aarch64-apple-clang-13.1.6-clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj.spec.json
  Download failed: HTTP Error 404: Not Found
==> [2022-08-02-21:31:02.592172] Did not find darwin-monterey-aarch64-apple-clang-13.1.6-clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj.spec.json on https://mirror.spack.io/build_cache/darwin-monterey-aarch64-apple-clang-13.1.6-clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj.spec.yaml
  Download failed: HTTP Error 404: Not Found
==> [2022-08-02-21:31:02.594142] No binary for clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj found: installing from source
==> [2022-08-02-21:31:02.594874] Creating the installation directory /Users/rountree4/spack/user_cache/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj
==> [2022-08-02-21:31:02.614958] '/usr/bin/git' '-C' '/Users/rountree4/spack' 'rev-parse' 'HEAD'
==> [2022-08-02-21:31:02.990406] Error: NameError: name 'directory' is not defined

/Users/rountree4/spack/var/spack/repos/builtin/packages/python/package.py:1115, in headers:
       1112                    break
       1113            else:
       1114                msg = "Unable to locate {} headers in {}"
  >>   1115                raise spack.error.NoHeadersError(msg.format(self.name, directory))
       1116
       1117        headers.directories = [os.path.dirname(config_h)]
       1118        return headers


Traceback (most recent call last):
  File "/Users/rountree4/spack/lib/spack/spack/build_environment.py", line 1058, in _setup_pkg_and_run
    kwargs["env_modifications"] = setup_package(
  File "/Users/rountree4/spack/lib/spack/spack/build_environment.py", line 804, in setup_package
    set_wrapper_variables(pkg, env_mods)
  File "/Users/rountree4/spack/lib/spack/spack/build_environment.py", line 462, in set_wrapper_variables
    update_compiler_args_for_dep(dspec)
  File "/Users/rountree4/spack/lib/spack/spack/build_environment.py", line 456, in update_compiler_args_for_dep
    _prepend_all(include_dirs, query.headers.directories)
  File "/Users/rountree4/spack/lib/spack/spack/spec.py", line 1082, in __get__
    value = f()
  File "/Users/rountree4/spack/lib/spack/spack/spec.py", line 1071, in <lambda>
    callbacks_chain.append(lambda: getattr(pkg, self.attribute_name))
  File "/Users/rountree4/spack/var/spack/repos/builtin/packages/python/package.py", line 1116, in headers
NameError: name 'directory' is not defined
==> [2022-08-02-21:31:02.990728] Flagging clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj as failed: NameError: name 'directory' is not defined
==> [2022-08-02-21:31:02.996661] Creating stage lock spack-stage-clingo-bootstrap-spack-2dqk5k7hjlv4rmyx7swunzoysdmkrixj
==> [2022-08-02-21:31:02.999180] Reading config file /Users/rountree4/spack/etc/spack/defaults/config.yaml
Traceback (most recent call last):
  File "/Users/rountree4/spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/Users/rountree4/spack/lib/spack/spack/main.py", line 961, in main
    return _main(argv)
  File "/Users/rountree4/spack/lib/spack/spack/main.py", line 916, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/Users/rountree4/spack/lib/spack/spack/main.py", line 944, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/Users/rountree4/spack/lib/spack/spack/main.py", line 608, in _invoke_command
    return_val = command(parser, args)
  File "/Users/rountree4/spack/lib/spack/spack/cmd/spec.py", line 104, in spec
    concretized_specs = spack.cmd.parse_specs(args.specs, concretize=True)
  File "/Users/rountree4/spack/lib/spack/spack/cmd/__init__.py", line 227, in parse_specs
    spec.concretize(tests=tests)  # implies normalize
  File "/Users/rountree4/spack/lib/spack/spack/spec.py", line 2924, in concretize
    self._new_concretize(tests)
  File "/Users/rountree4/spack/lib/spack/spack/spec.py", line 2898, in _new_concretize
    solver = spack.solver.asp.Solver()
  File "/Users/rountree4/spack/lib/spack/spack/solver/asp.py", line 2172, in __init__
    self.driver = PyclingoDriver()
  File "/Users/rountree4/spack/lib/spack/spack/solver/asp.py", line 518, in __init__
    bootstrap_clingo()
  File "/Users/rountree4/spack/lib/spack/spack/solver/asp.py", line 498, in bootstrap_clingo
    spack.bootstrap.ensure_clingo_importable_or_raise()
  File "/Users/rountree4/spack/lib/spack/spack/bootstrap.py", line 774, in ensure_clingo_importable_or_raise
    ensure_module_importable_or_raise(module="clingo", abstract_spec=clingo_root_spec())
  File "/Users/rountree4/spack/lib/spack/spack/bootstrap.py", line 550, in ensure_module_importable_or_raise
    raise ImportError(msg)
ImportError: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %apple-clang target=aarch64" due to the following failures:
github-actions-v0.1 raised ValueError: source is not trusted
  File "/Users/rountree4/spack/lib/spack/spack/bootstrap.py", line 532, in ensure_module_importable_or_raise
    source_is_enabled_or_raise(current_config)
  File "/Users/rountree4/spack/lib/spack/spack/bootstrap.py", line 464, in source_is_enabled_or_raise
    raise ValueError("source is not trusted")

spack-install raised InstallError: Terminating after first install failure: NameError: name 'directory' is not defined
  File "/Users/rountree4/spack/lib/spack/spack/bootstrap.py", line 535, in ensure_module_importable_or_raise
    if b.try_import(module, abstract_spec):
  File "/Users/rountree4/spack/lib/spack/spack/bootstrap.py", line 417, in try_import
    concrete_spec.package.do_install(fail_fast=True)
  File "/Users/rountree4/spack/lib/spack/spack/package_base.py", line 1928, in do_install
    builder.install()
  File "/Users/rountree4/spack/lib/spack/spack/installer.py", line 1718, in install
    raise InstallError("{0}: {1}".format(fail_fast_err, str(exc)))
```

### Information on your system

```
rountree4@exegete spack % spack debug report                     
* **Spack:** 0.19.0.dev0 (4d2ed35552697cc6a41e92395cfdc0edc60a0733)
* **Python:** 3.8.9
* **Platform:** darwin-monterey-m1
* **Concretizer:** clingo
```

also

```
rountree4@exegete ~ % spack debug report                                                                                
* **Spack:** 0.19.0.dev0 (b3f221185e1d4c56bbb38c8b7669beec2885bb06)
* **Python:** 3.8.9
* **Platform:** darwin-monterey-m1
* **Concretizer:** clingo
```

```
rountree4@exegete spack % spack compiler list
==> Available compilers
-- apple-clang monterey-aarch64 ---------------------------------
apple-clang@13.1.6
```


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
It seems there are two issues here:
1. Python headers are not found by Spack
2. There's a reference to an unset variable during error reporting
This looks relevant:
```
commit 90dbfea895454a22604832ddf8e6e6b8169489cf
Author: Adam J. Stewart <ajstewart426@gmail.com>
Date:   Mon Aug 1 08:48:07 2022 -0700

    python: fix clingo bootstrapping on Apple M1/M2 (#31792)
```

Part of that commit:
```
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
index 79dc35fafc..9f3ffa92d9 100644
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -961,6 +961,7 @@ def config_vars(self):
                 # get_config_vars
                 "BINDIR": self.prefix.bin,
                 "CC": "cc",
+                "CONFINCLUDEPY": self.prefix.include.join("python{}").format(version),
                 "CXX": "c++",
                 "INCLUDEPY": self.prefix.include.join("python{}").format(version),
                 "LIBDEST": self.prefix.lib.join("python{}").format(version),
@@ -1098,15 +1099,17 @@ def libs(self):
 
     @property
     def headers(self):
-        directory = self.config_vars["include"]
+        # Location where pyconfig.h is _supposed_ to be
         config_h = self.config_vars["config_h_filename"]
-
         if os.path.exists(config_h):
             headers = HeaderList(config_h)
         else:
-            headers = find_headers("pyconfig", directory)
-            if headers:
-                config_h = headers[0]
+            # If not, one of these config vars should contain the right directory
+            for var in ["INCLUDEPY", "CONFINCLUDEPY"]:
+                headers = find_headers("pyconfig", self.config_vars[var])
+                if headers:
+                    config_h = headers[0]
+                    break
             else:
                 msg = "Unable to locate {} headers in {}"
                 raise spack.error.NoHeadersError(msg.format(self.name, directory))
```

If I'm reading that correctly, the `else` ought to be indented to line up under `if headers:` but that means `CONFINCLUDEPY` is never checked (the `else` trips if `INCLUDEPY` isn't set.

After fooling around a bit with the code it doesn't look like either `CONFINCLUDEPY` or `INCLUDEPY` works.
>If I'm reading that correctly, the else ought to be indented to line up under if headers: but that means CONFINCLUDEPY is never checked (the else trips if INCLUDEPY isn't set.

`else` is a valid keyword to be used in a for construct, as it is here. It is run after all the iterations and usually makes sense in a loop that breaks if some condition is met.

Here the issue seems to be:
```diff
-        directory = self.config_vars["include"]
+        # Location where pyconfig.h is _supposed_ to be
```
because the `directory` variable is used in the error message under the `else`. Anyhow, the real issue for failing to bootstrap is that headers are not found.
I'll fix the missing variable, that's easy.

For the headers not being found, can you post the output of `python3 -m sysconfig`? Also, can you look for a `pyconfig.h` file somewhere in the directories listed? It's usually under `INCLUDEPY` but it seems like it's not being found for some reason.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/clingo/package.py b/var/spack/repos/builtin/packages/clingo/package.py
--- a/var/spack/repos/builtin/packages/clingo/package.py
+++ b/var/spack/repos/builtin/packages/clingo/package.py
@@ -74,8 +74,13 @@ def cmake_python_hints(self):
         """
         python = self.spec["python"]
         return [
-            self.define("Python_EXECUTABLE", python.command),
+            self.define("Python_EXECUTABLE", python.command.path),
             self.define("Python_INCLUDE_DIR", python.headers.directories[0]),
+            self.define("Python_LIBRARIES", python.libs[0]),
+            # XCode command line tools on macOS has no python-config executable, and
+            # CMake assumes you have python 2 if it does not find a python-config,
+            # so we set the version explicitly so that it's passed to FindPython.
+            self.define("CLINGO_PYTHON_VERSION", python.version.up_to(2)),
         ]
 
     @property
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -1044,6 +1044,12 @@ def find_library(self, library):
         win_bin_dir = self.config_vars["BINDIR"]
 
         directories = [libdir, libpl, frameworkprefix, macos_developerdir, win_bin_dir]
+
+        # The Python shipped with Xcode command line tools isn't in any of these locations
+        for subdir in ["lib", "lib64"]:
+            directories.append(os.path.join(self.config_vars["base"], subdir))
+
+        directories = dedupe(directories)
         for directory in directories:
             path = os.path.join(directory, library)
             if os.path.exists(path):
@@ -1051,52 +1057,64 @@ def find_library(self, library):
 
     @property
     def libs(self):
-        # The +shared variant isn't always reliable, as `spack external find`
-        # currently can't detect it. If +shared, prefer the shared libraries, but check
-        # for static if those aren't found. Vice versa for ~shared.
+        py_version = self.version.up_to(2)
 
-        # The values of LDLIBRARY and LIBRARY also aren't reliable. Intel Python uses a
+        # The values of LDLIBRARY and LIBRARY aren't reliable. Intel Python uses a
         # static binary but installs shared libraries, so sysconfig reports
-        # libpythonX.Y.a but only libpythonX.Y.so exists.
+        # libpythonX.Y.a but only libpythonX.Y.so exists. So we add our own paths, too.
         shared_libs = [
             self.config_vars["LDLIBRARY"],
-            "libpython{}.{}".format(self.version.up_to(2), dso_suffix),
+            "libpython{}.{}".format(py_version, dso_suffix),
         ]
         static_libs = [
             self.config_vars["LIBRARY"],
-            "libpython{}.a".format(self.version.up_to(2)),
+            "libpython{}.a".format(py_version),
         ]
+
+        # The +shared variant isn't reliable, as `spack external find` currently can't
+        # detect it. If +shared, prefer the shared libraries, but check for static if
+        # those aren't found. Vice versa for ~shared.
         if "+shared" in self.spec:
-            libraries = shared_libs + static_libs
+            candidates = shared_libs + static_libs
         else:
-            libraries = static_libs + shared_libs
-        libraries = dedupe(libraries)
+            candidates = static_libs + shared_libs
+        candidates = dedupe(candidates)
 
-        for library in libraries:
-            lib = self.find_library(library)
+        for candidate in candidates:
+            lib = self.find_library(candidate)
             if lib:
                 return lib
 
-        msg = "Unable to locate {} libraries in {}"
-        libdir = self.config_vars["LIBDIR"]
-        raise spack.error.NoLibrariesError(msg.format(self.name, libdir))
+        raise spack.error.NoLibrariesError(
+            "Unable to find {} libraries with the following names:".format(self.name),
+            "\n".join(candidates),
+        )
 
     @property
     def headers(self):
-        # Location where pyconfig.h is _supposed_ to be
-        config_h = self.config_vars["config_h_filename"]
-        if os.path.exists(config_h):
-            headers = HeaderList(config_h)
+        # Locations where pyconfig.h could be
+        # This varies by system, especially on macOS where the command line tools are
+        # installed in a very different directory from the system python interpreter.
+        py_version = str(self.version.up_to(2))
+        candidates = [
+            os.path.dirname(self.config_vars["config_h_filename"]),
+            self.config_vars["INCLUDEPY"],
+            self.config_vars["CONFINCLUDEPY"],
+            os.path.join(self.config_vars["base"], "include", py_version),
+            os.path.join(self.config_vars["base"], "Headers"),
+        ]
+        candidates = list(dedupe(candidates))
+
+        for directory in candidates:
+            headers = find_headers("pyconfig", directory)
+            if headers:
+                config_h = headers[0]
+                break
         else:
-            # If not, one of these config vars should contain the right directory
-            for var in ["INCLUDEPY", "CONFINCLUDEPY"]:
-                headers = find_headers("pyconfig", self.config_vars[var])
-                if headers:
-                    config_h = headers[0]
-                    break
-            else:
-                msg = "Unable to locate {} headers in {}"
-                raise spack.error.NoHeadersError(msg.format(self.name, directory))
+            raise spack.error.NoHeadersError(
+                "Unable to locate {} headers in any of these locations:".format(self.name),
+                "\n".join(candidates),
+            )
 
         headers.directories = [os.path.dirname(config_h)]
         return headers


Test patch: