Repository: spack/spack

Issue:
ROOT + Python/libxml2: Maximum Recursion Depth
Trying to install `root` via
```bash
$ spack spec root
```
leads to
```
Input spec
--------------------------------
root

Concretized
--------------------------------
==> Error: maximum recursion depth exceeded
```

The problem is gone when using `root ~python`.


---

### Steps to reproduce the issue

```console
$ spack install root
```

### Platform and user environment

Please report your OS here:
```commandline
$ uname -a 
Linux axel-work 5.0.0-29-generic #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -d
Description:	Ubuntu 18.04.3 LTS
``` 

### Additional information

Spack version: `develop` as of ae6229dee20da04b2120dc97cdfae800c136e42a

Note 1: `root +x` (default) does not build (seen via `root ~xml`).
Note 2: `root ~x` does not build unless I also deactivate `~opengl` (default: `+opengl`).
remove gettext dependency from python
Fixes https://github.com/spack/spack/issues/13637

Python depends on libintl.h which is provided by the 'gettext' package, but adding 'gettext' as a dependency creates a cyclic dependency. It should be investigated further how to resolve this conflict. In the meantime this PR removes it because it hasn't yet caused any build failures to build without it.


Discussion:
The problem is rooted, :trumpet: in
```py
    depends_on('libxml2+python',   when='+xml+python')
    depends_on('libxml2~python',   when='+xml~python')
```

work-around: `root~xml`
EDIT: I can reproduce this myself with on MacOS and a nearly-clean environment (I can try with a completely clean environment tomorrow).
@scheibelp If it starts to look like a problem with the recipe, or you need help isolating a regression, tag me in.
@adamjstewart @tgamblin @scheibelp Could this be related to root cause of https://github.com/spack/spack/issues/13555#issuecomment-549430024?
Nominally I can trace this issue back to https://github.com/spack/spack/pull/10944 (this commit fails and the prior one succeeds). That being said I don't specifically blame that PR: Spack shouldn't fail like this regardless of what constraints exist. I'll be looking into why that is causing an issue today.
Of all the changes in that PR, the one that appears to cause this issue is adding `gettext` as a dependency to Python (I can get `root` to concretize if I remove that dependency). I need to investigate why that is (and similar to what I mentioned above, I don't consider this the core cause of the issue).
There appears to be a cyclic dependency:

`libxml2+python -> python -> gettext -> libxml2`

`root` requests `libxml2+python` by default, which is why https://github.com/spack/spack/issues/13637#issuecomment-551356181 works. Spack should do a better job of reporting this cyclic dependency. Figuring out exactly why it doesn't detect and report this better requires more work.

That being said, overall Spack can't actually resolve the constraints as they are currently specified across the involved packages: one of them is going to have to be more specific (e.g. perhaps Python should explicitly request `gettext` without `libxml2` support).
It would appear that in RHEL-7 and friends, system `gettext`is compiled _with_ `libxml2` support, and system `python` is compiled without explicit `gettext` support, but on GNU systems there's a good chance that `gettext()` is pulled straight from `glibc`. It would seem to be sensible to follow the same prescription in the `python` build recipe as the respective systems do. I would note that neither homebrew's (macOS), our own (Fermilab's) build recipe (macOS, RHEL-ish, Ubuntu LTS), nor RedHat's own SPEC file refer to the `gettext` package at all (although RedHat's `python` package does provide a `gettext.py`).

TL;DR: IMO the dependency of `python` on `gettext` should be removed in order to avoid this circular dependency. The non-detection of same by Spack's concretizer is a completely different problem that I am glad is not mine 😉.
As a minor point, there is no reason I can think of for the `root` recipe to require a dependency on `libxml2~python` when `~python`: it should just depend upon `libxml2` without qualification, and take what it gets.
> TL;DR: IMO the dependency of python on gettext should be removed in order to avoid this circular dependency.

(EDIT: PR is planned for Monday 11/11)

I agree. I'll create a PR to remove it. That being said I think I'll also want to investigate it more to understand the details.

> It would appear that in RHEL-7 and friends, system gettext is compiled with libxml2 support, and system python is compiled without explicit gettext support, but on GNU systems there's a good chance that gettext() is pulled straight from glibc

#10944 added `gettext` to python on account of https://github.com/spack/spack/pull/10944#discussion_r270608085: the author noticed a configure check in the `python` package which referred to it. However, the author also noted that they originally added this because they encountered a build failure that they could not reproduce later, so currently I don't think there's evidence that removing the `gettext` dependency from `python` will cause problems.

Do you know if Python *can* be compiled with explicit `gettext` support? There didn't appear to be a flag to enable/disable it, so it seemed mandatory.
Usage of `libintl.h` ([gettext](https://www.gnu.org/software/gettext/manual/html_node/Overview.html#Overview)) in `CPython`:
  https://github.com/python/cpython/search?q=libintl&unscoped_q=libintl

and [warnings in its README on Mac](https://github.com/python/cpython/blob/v3.8.0/Mac/README.rst#configure-warnings)

> This almost always means you are trying to build a universal binary for Python and have libraries in /usr/local that don't contain the required architectures. Temporarily move /usr/local aside to finish the build.

[configure](https://github.com/python/cpython/blob/v3.8.0/configure.ac#L2822-L2826) scripts.

Seams to be used for i18n of CPython itself, e.g. in [`Modules/_localemodule.c`](https://github.com/python/cpython/blob/v3.8.0/Modules/_localemodule.c).
Now hold your horses, [gettext](https://www.gnu.org/software/gettext/manual/html_node/Python.html#Python) can itself also provide a python module(?)/dependency :laughing: 
(v0.11.1+)
(But we are not pulling that in our package.)

Update: checking the `gettext` sources this looks more like an example to me, not a full Python module build.
That might mean it could require python as a build dependency, and once we can support multiple instances of packages as build dependencies I don't think there would be an issue with cyclic dependencies. Although for now luckily no one has needed that functionality for `gettext` in Spack.

It's unfortunate if the only currently-supported mechanism for avoiding `libintl` warnings is to move `/usr/local/`. Perhaps long-term (if the temporary issue reported in #10944 crops up again) the solution is to patch the build system to remove the libintl check.
After doing a full build of python with the `depends_on('gettext')` removed, I can confirm that the only mention of "gettext" in the build log is the provided `gettext.py`. We do have:
```
checking libintl.h usability... no
checking libintl.h presence... no
checking for libintl.h... no
```
but that appears to have no discernible effect on the build.
Okay, just be aware that gettext provides `libintl.h` and disabling it might affect internationalization (multi-language) support of the cpython build for some users.

@chissg I guess you build on osx with clang [as in the original comment](https://github.com/spack/spack/pull/10944#discussion_r270608085)?
In my defaults (`%gcc@8.3.0 arch=linux-ubuntu18.04-skylake`), with `gettext` removed, Python 2.7.15 and 3.8.0 writes:
```
checking libintl.h usability... yes
checking libintl.h presence... yes
checking for libintl.h... yes
```
but who knows from where this is picked up, I have one in `/usr/include/libintl.h` by default from [libc6-dev](https://packages.ubuntu.com/search?searchon=contents&keywords=libintl.h&mode=exactfilename&suite=disco&arch=any). Maybe we can set a `--without-gettext`/`--without-libintl` flag or something to `configure` to have control (at least on osx).
I'll be adding a PR for this shortly but anticipate merging it after 11/25 to help our Spack tutorial at SC run smoothly (removing `gettext` from `python` will change the hashes of quite a few packages including those that are installed at the tutorial).


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/libxml2/package.py b/var/spack/repos/builtin/packages/libxml2/package.py
--- a/var/spack/repos/builtin/packages/libxml2/package.py
+++ b/var/spack/repos/builtin/packages/libxml2/package.py
@@ -27,7 +27,8 @@ class Libxml2(AutotoolsPackage):
     depends_on('zlib')
     depends_on('xz')
 
-    depends_on('python+shared', when='+python')
+    # avoid cycle dependency for concretizer
+    depends_on('python+shared~libxml2', when='+python')
     extends('python', when='+python',
             ignore=r'(bin.*$)|(include.*$)|(share.*$)|(lib/libxml2.*$)|'
             '(lib/xml2.*$)|(lib/cmake.*$)')
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -70,6 +70,10 @@ class Python(AutotoolsPackage):
 
     extendable = True
 
+    # Variants to avoid cyclical dependencies for concretizer
+    variant('libxml2', default=False,
+            description='Use a gettext library build with libxml2')
+
     variant(
         'debug', default=False,
         description="debug build with extra checks (this is high overhead)"
@@ -116,7 +120,8 @@ class Python(AutotoolsPackage):
     variant('tix',      default=False, description='Build Tix module')
 
     depends_on('pkgconfig@0.9.0:', type='build')
-    depends_on('gettext')
+    depends_on('gettext +libxml2', when='+libxml2')
+    depends_on('gettext ~libxml2', when='~libxml2')
 
     # Optional dependencies
     # See detect_modules() in setup.py for details


Test patch: