Repository: spack/spack

Issue:
Unsatisfiable constraint when no explicit package version given
Installing `flux-core` with an explicit version (i.e., `flux-core@0.9.0`, `flux-core@0.10.0`, `flux-core@master`) works perfectly.  Attempting to install without an explicit version (i.e., `spack install flux-core`) fails when spack cannot seem to satisfy the lua dependency.  It appears as though the concretizer just grabs tha latest version of lua (5.3.X) and then some other verification pass realizes that this doesn't meet the specification for the latest version of flux-core (flux-core@0.10.0 and lua@5.1:5.2.99).

My guess is that this is either:
A) improper specification of dependencies in the flux-core package
B) a bug in the concretizer

Flux's lua dependency specification currently looks like:
```python
    depends_on("lua", type=('build', 'run', 'link'))
    depends_on("lua@5.1:5.1.99", when="@0.1.0:0.9.0")
    depends_on("lua@5.1:5.2.99", when="@0.10.0:,master")
```

Does the `when` argument only apply when a user explicitly adds that variant/customization in the command?

### Steps to reproduce the issue & Error Message

```console
$ spack -d install flux-core
<snip some lines about reading configs and getting locks>
--------------------------------
flux-core

Concretized
--------------------------------
==> Error: An unsatisfiable version constraint has been detected for spec:

    lua@5.3.4%gcc@8.2.0 arch=linux-rhel7-x86_64 
        ^ncurses@6.1%gcc@8.2.0~symlinks~termlib arch=linux-rhel7-x86_64 
            ^pkgconf@1.5.4%gcc@8.2.0 arch=linux-rhel7-x86_64 
        ^readline@7.0%gcc@8.2.0 arch=linux-rhel7-x86_64 
        ^unzip@6.0%gcc@8.2.0 arch=linux-rhel7-x86_64 


while trying to concretize the partial spec:

    flux-core@0.10.0%gcc@8.2.0~cuda~docs arch=linux-rhel7-x86_64 
        ^czmq@4.1.1%gcc@8.2.0 arch=linux-rhel7-x86_64 
            ^autoconf@2.69%gcc@8.2.0 arch=linux-rhel7-x86_64 
                ^m4@1.4.18%gcc@8.2.0+sigsegv arch=linux-rhel7-x86_64 
                    ^libsigsegv
                ^perl@5.26.2%gcc@8.2.0+cpanm+shared+threads arch=linux-rhel7-x86_64 
                    ^gdbm@1.18.1%gcc@8.2.0 arch=linux-rhel7-x86_64 
                        ^readline@7.0%gcc@8.2.0 arch=linux-rhel7-x86_64 
                            ^ncurses@6.1%gcc@8.2.0~symlinks~termlib arch=linux-rhel7-x86_64 
                                ^pkgconf@1.5.4%gcc@8.2.0 arch=linux-rhel7-x86_64 
            ^automake@1.16.1%gcc@8.2.0 arch=linux-rhel7-x86_64 
            ^libtool@2.4.6%gcc@8.2.0 arch=linux-rhel7-x86_64 
            ^libuuid@1.0.3%gcc@8.2.0 arch=linux-rhel7-x86_64 
            ^zeromq@4.2.5%gcc@8.2.0+libsodium arch=linux-rhel7-x86_64 
                ^libsodium
        ^hwloc@1.11.9%gcc@8.2.0~cairo~cuda+libxml2+pci+shared arch=linux-rhel7-x86_64 
            ^libpciaccess
                ^util-macros
            ^libxml2
                ^xz
                ^zlib@1.2.11%gcc@8.2.0+optimize+pic+shared arch=linux-rhel7-x86_64 
            ^numactl
        ^jansson@2.9%gcc@8.2.0 build_type=RelWithDebInfo +shared arch=linux-rhel7-x86_64 
            ^cmake@3.13.0%gcc@8.2.0~doc+ncurses+openssl+ownlibs~qt arch=linux-rhel7-x86_64 
                ^openssl@1.1.1%gcc@8.2.0+systemcerts arch=linux-rhel7-x86_64 
        ^lua-luaposix@33.4.0%gcc@8.2.0 arch=linux-rhel7-x86_64 
            ^lua@5.3.4%gcc@8.2.0 arch=linux-rhel7-x86_64 
                ^unzip@6.0%gcc@8.2.0 arch=linux-rhel7-x86_64 
        ^munge@0.5.13%gcc@8.2.0 arch=linux-rhel7-x86_64 
            ^libgcrypt@1.8.1%gcc@8.2.0 arch=linux-rhel7-x86_64 
                ^libgpg-error@1.27%gcc@8.2.0 arch=linux-rhel7-x86_64 
        ^python@2.7.15%gcc@8.2.0+dbm~optimizations+pic+pythoncmd+shared~tk~ucs4 arch=linux-rhel7-x86_64 
            ^bzip2@1.0.6%gcc@8.2.0+shared arch=linux-rhel7-x86_64 
                ^diffutils@3.6%gcc@8.2.0 arch=linux-rhel7-x86_64 
            ^sqlite@3.25.3%gcc@8.2.0~functions arch=linux-rhel7-x86_64 


flux-core requires lua version 5.1:5.2.99, but spec asked for 5.3.4
Traceback (most recent call last):
  File "/g/g0/herbein1/Repositories/spack/lib/spack/spack/main.py", line 669, in main
    return _invoke_command(command, parser, args, unknown)
  File "/g/g0/herbein1/Repositories/spack/lib/spack/spack/main.py", line 440, in _invoke_command
    return_val = command(parser, args)
  File "/g/g0/herbein1/Repositories/spack/lib/spack/spack/cmd/spec.py", line 72, in spec
    spec.concretize()
  File "/g/g0/herbein1/Repositories/spack/lib/spack/spack/spec.py", line 1865, in concretize
    user_spec_deps=user_spec_deps),
  File "/g/g0/herbein1/Repositories/spack/lib/spack/spack/spec.py", line 2288, in normalize
    visited, all_spec_deps, provider_index, tests)
  File "/g/g0/herbein1/Repositories/spack/lib/spack/spack/spec.py", line 2227, in _normalize_helper
    dep, visited, spec_deps, provider_index, tests)
  File "/g/g0/herbein1/Repositories/spack/lib/spack/spack/spec.py", line 2168, in _merge_dependency
    changed |= spec_deps[dep.name].constrain(dep)
  File "/g/g0/herbein1/Repositories/spack/lib/spack/spack/spec.py", line 2361, in constrain
    raise UnsatisfiableVersionSpecError(self.versions, other.versions)
spack.spec.UnsatisfiableVersionSpecError: An unsatisfiable version constraint has been detected for spec:
<snip repeated spec>
```

### Information on your system

This includes:

 1. TOSS3 system at LLNL


Discussion:
That is a known shortcoming of the current concretiser, for now we need to provide it with enough hints such that it succeeds. Last I heard was that the new one was supposed to be merged in September (which obviously didn't happen), but maybe @scheibelp knows more
@SteVwonder @tgamblin Are we giving away a mug or something for the 10k issue / PR ? :laughing: 
Or maybe a new concretizer? :angel: 
> Are we giving away a mug or something for the 10k issue

:laughing: I'll gladly accept and cherish any and all free spack merch sent my way

> or now we need to provide it with enough hints such that it succeeds

Do you have any suggested hints or previous examples that I can look at?

Although not perfect, I suppose I could add version constraints to the `when`-less lua under the assumption that v0.10 is being used when no explicit flux-core version is provided.  Specifically:
```python
    depends_on("lua@5.1:5.2.99", type=('build', 'run', 'link'))
    depends_on("lua@5.1:5.1.99", when="@0.1.0:0.9.0")
    depends_on("lua@5.1:5.2.99", when="@0.10.0:,master")
```
Thoughts?
For us we have added a number of entries in `etc/spack/packages.yaml`, so in your case you could add a 
```
packages:
  lua:
    version: [5.1.x]
```
That should make it prefer a compatible lua version. The problem occurs because the concretiser has to autocompletes things when it does not yet know enough and then does not backtrack far enough.
That would work for me specifically, but we are pushing spack as a good, general way to install and use flux.  So having `spack install flux-core` work "out-of-the-box" is really important to us.  I'm hoping to come up with a solution (even if its a bit hacky) that would work for anyone that clones spack.

Is there a way to provide hints from within the `package.py`?  Maybe the flux package could inspect the spec to determine if an explicit version was given, and if not, set the lua version constraints (i.e., a hint).

> The problem occurs because the concretiser has to autocompletes things when it does not yet know enough and then does not backtrack far enough.

This is the part I don't quite grok.  Does the concretiser start by autocompleting the root package and working its way down?  If so, it would start by setting flux-core to @0.10.0, then when it goes to select the version for lua, it should be able to apply the flux-core@0.10.0 specific constraints (i.e., lua 5.1.X-5.2.X).  Since that doesn't happen, I'm guessing the concretizer starts at the leaves of the dependency tree and works it's way to the root?  Is there a way to set/change the backtracking depth?

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/flux-core/package.py b/var/spack/repos/builtin/packages/flux-core/package.py
--- a/var/spack/repos/builtin/packages/flux-core/package.py
+++ b/var/spack/repos/builtin/packages/flux-core/package.py
@@ -32,11 +32,13 @@ class FluxCore(AutotoolsPackage):
     depends_on("czmq@3.0.1:", when="@0.7:,master")
     depends_on("hwloc@1.11.1:1.99")
     depends_on("hwloc +cuda", when='+cuda')
-    depends_on("lua", type=('build', 'run', 'link'))
+    # Provide version hints for lua so that the concretizer succeeds when no
+    # explicit flux-core version is given. See issue #10000 for details
+    depends_on("lua@5.1:5.2.99", type=('build', 'run', 'link'))
     depends_on("lua@5.1:5.1.99", when="@0.1.0:0.9.0")
     depends_on("lua@5.1:5.2.99", when="@0.10.0:,master")
     depends_on("lua-luaposix")
-    depends_on("munge")
+    depends_on("munge", when="@0.1.0:0.10.0")
     depends_on("libuuid")
     depends_on("python", type=('build', 'run'))
     depends_on("python@2.7:2.99", when="@0.1.0:0.10.0")
diff --git a/var/spack/repos/builtin/packages/flux-sched/package.py b/var/spack/repos/builtin/packages/flux-sched/package.py
--- a/var/spack/repos/builtin/packages/flux-sched/package.py
+++ b/var/spack/repos/builtin/packages/flux-sched/package.py
@@ -19,6 +19,10 @@ class FluxSched(AutotoolsPackage):
     version('0.5.0', 'a9835c9c478aa41123a4e12672500052228aaf1ea770f74cb0901dbf4a049bd7d329e99d8d3484e39cfed1f911705030b2775dcfede39bc8bea59c6afe2549b1')
     version('0.4.0', '82732641ac4594ffe9b94ca442a99e92bf5f91bc14745af92203a887a40610dd44edda3ae07f9b6c8d63799b2968d87c8da28f1488edef1310d0d12be9bd6319')
 
+    # Avoid the infinite symlink issue
+    # This workaround is documented in PR #3543
+    build_directory = 'spack-build'
+
     variant('cuda', default=False, description='Build dependencies with support for CUDA')
 
     depends_on("boost+graph", when='@0.5.0:,master')
@@ -52,7 +56,10 @@ def setup(self):
 
     def autoreconf(self, spec, prefix):
         self.setup()
-        if not os.path.exists('configure'):
+        if os.path.exists(self.configure_abs_path):
+            return
+        # make sure configure doesn't get confused by the staging symlink
+        with working_dir(self.configure_directory):
             # Bootstrap with autotools
             bash = which('bash')
             bash('./autogen.sh')


Test patch: