Repository: spack/spack

Issue:
Installation issue: neovim doesn't concretize
### Steps to reproduce the issue

```console
$ spack spec -I neovim
==> Error:
```


### Error message

<details><summary>Error message</summary>

<pre>
The error message is empty as noted in the output of spack spec above
</pre></details>


### Information on your system

* **Spack:** 0.22.0.dev0 (9e0720207afb5b0fed4035dc1dbf214a1eddfca1)
* **Python:** 3.9.15
* **Platform:** linux-ubuntu20.04-skylake
* **Concretizer:** clingo

### Additional information

Since the spec doesn't even concretize, no build files were generated.  I have confirmed this fails on `releases/v0.21`, but succeeds on `releases/v0.20` (i.e. this spec stops concretizing after 0.20).  @albestro @trws I am mentioning you as per the instructions in the issue template, but I'm actually unsure whether this is a bug in the neovim package or the concretizer.

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
Can you use `spack --backtrace spec -I neovim`?
@haampie see here:
```
$ spack --backtrace spec -I neovim
Traceback (most recent call last):
  File "/home/tasmith/temp_spack/spack/bin/spack", line 52, in <module>
    sys.exit(main())
  File "/home/tasmith/temp_spack/spack/lib/spack/spack_installable/main.py", line 37, in main
    sys.exit(spack.main.main(argv))
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/main.py", line 1066, in main
    return _main(argv)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/main.py", line 1019, in _main
    return finish_parse_and_run(parser, cmd_name, args, env_format_error)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/main.py", line 1049, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/main.py", line 648, in _invoke_command
    return_val = command(parser, args)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/cmd/spec.py", line 101, in spec
    concretized_specs = spack.cmd.parse_specs(args.specs, concretize=True)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/cmd/__init__.py", line 218, in parse_specs
    spec.concretize(tests=tests)  # implies normalize
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/spec.py", line 2987, in concretize
    self._new_concretize(tests)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/spec.py", line 2959, in _new_concretize
    result = solver.solve([self], tests=tests, allow_deprecated=allow_deprecated)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/solver/asp.py", line 3257, in solve
    result, _, _ = self.driver.solve(
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/solver/asp.py", line 1001, in solve
    answers = builder.build_specs(spec_attrs)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/solver/asp.py", line 3099, in build_specs
    root._finalize_concretization()
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/spec.py", line 3064, in _finalize_concretization
    spec._cached_hash(ht.package_hash, force=True)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/spec.py", line 1872, in _cached_hash
    hash_string = self.spec_hash(hash)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/spec.py", line 1843, in spec_hash
    return hash.override(self)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/hash_types.py", line 53, in _content_hash_override
    pkg = pkg_cls(spec)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/package_base.py", line 677, in __init__
    pkg_cls = spack.repo.PATH.get_pkg_class(self.extendee_spec.name)
  File "/home/tasmith/temp_spack/spack/lib/spack/spack/package_base.py", line 1242, in extendee_spec
    assert len(deps) == 1
AssertionError
```
Hm, weird. Can you share the spec?

```diff
diff --git a/lib/spack/spack/package_base.py b/lib/spack/spack/package_base.py
index 416b16cefc..aec815c3e9 100644
--- a/lib/spack/spack/package_base.py
+++ b/lib/spack/spack/package_base.py
@@ -1234,6 +1234,8 @@ def extendee_spec(self):
         deps = []
 
         # If the extendee is in the spec's deps already, return that.
+        with open("spec.json", "w") as f:
+            self.spec.to_json(f)
         for dep in self.spec.traverse(deptype=("link", "run")):
             if dep.name in self.extendees:
                 deps.append(dep)
```


see here:
[spec.json](https://github.com/spack/spack/files/13550905/spec.json)

I encounter exactly this issue too since spack 0.21, with `spack spec`**`--fresh`**`neovim`, on ubuntu22 and rhel8.

FWIW, git-bisect points to 16fa3b9f077be62e62214585dee9f6dfda7f48ad
I was able to reproduce it with a minimal recipe, it seems to come from confusions about the lua-lang+luajit double-virtual resolution:

```py
from spack.package import *
class Testlua(BundlePackage):
    homepage = ""
    version("1")
    depends_on("lua-lang")
    depends_on("luajit")
    depends_on("lua-bitlib")
```


It seems related to this

https://github.com/spack/spack/blob/b70cb60b6581ee4b02f786de516aa6a24457c063/lib/spack/spack/package_base.py#L1267-L1269

Indeed, `lua-bitlib` seems to end up with

```
extendees {'lua': (lua, None), 'lua-luajit': (lua-luajit, None), 'lua-luajit-openresty': (lua-luajit-openresty, None)}
```

and two of them are found among the dependencies

```
[0] lua@=5.4.4%gcc@=11.2.0~pcfile+shared build_system=makefile fetcher=curl arch=cray-cnl7-haswell ^[deptypes=build,link] berkeley-db@=18.1.40%gcc@=11.2.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=cray-cnl7-haswell ^[deptypes=build,link] bzip2@=1.0.8%gcc@=11.2.0~debug~pic+shared build_system=generic arch=cray-cnl7-haswell ^[deptypes=build] ca-certificates-mozilla@=2023-05-30%gcc@=11.2.0 build_system=generic arch=cray-cnl7-haswell ^[deptypes=run] curl@=8.4.0%gcc@=11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=cray-cnl7-haswell ^[deptypes=build] diffutils@=3.9%gcc@=11.2.0 build_system=autotools arch=cray-cnl7-haswell ^[deptypes=link] gcc-runtime@=11.2.0%gcc@=11.2.0 build_system=generic arch=cray-cnl7-haswell ^[deptypes=build,link] gdbm@=1.23%gcc@=11.2.0 build_system=autotools arch=cray-cnl7-haswell ^[deptypes=build] gmake@=4.4.1%gcc@=11.2.0~guile build_system=generic arch=cray-cnl7-haswell ^[deptypes=build,link virtuals=iconv] libiconv@=1.17%gcc@=11.2.0 build_system=autotools libs=shared,static arch=cray-cnl7-haswell ^[deptypes=build,link] ncurses@=6.4%gcc@=11.2.0~symlinks+termlib abi=none build_system=autotools arch=cray-cnl7-haswell ^[deptypes=build,link] nghttp2@=1.57.0%gcc@=11.2.0 build_system=autotools arch=cray-cnl7-haswell ^[deptypes=build,link] openssl@=3.1.3%gcc@=11.2.0~docs+shared build_system=generic certs=mozilla arch=cray-cnl7-haswell ^[deptypes=build] perl@=5.38.0%gcc@=11.2.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=cray-cnl7-haswell ^[deptypes=build virtuals=pkgconfig] pkgconf@=1.9.5%gcc@=11.2.0 build_system=autotools arch=cray-cnl7-haswell ^[deptypes=build,link] readline@=8.2%gcc@=11.2.0 build_system=autotools patches=bbf97f1 arch=cray-cnl7-haswell ^[deptypes=build,run] unzip@=6.0%gcc@=11.2.0 build_system=makefile arch=cray-cnl7-haswell ^[deptypes=build,link virtuals=zlib-api] zlib-ng@=2.1.5%gcc@=11.2.0+compat+opt build_system=autotools arch=cray-cnl7-haswell
[1] lua-luajit-openresty@=2.1-20230410%gcc@=11.2.0+lualinks build_system=makefile fetcher=curl arch=cray-cnl7-haswell ^[deptypes=build,link] berkeley-db@=18.1.40%gcc@=11.2.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=cray-cnl7-haswell ^[deptypes=build,link] bzip2@=1.0.8%gcc@=11.2.0~debug~pic+shared build_system=generic arch=cray-cnl7-haswell ^[deptypes=build] ca-certificates-mozilla@=2023-05-30%gcc@=11.2.0 build_system=generic arch=cray-cnl7-haswell ^[deptypes=run] curl@=8.4.0%gcc@=11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=cray-cnl7-haswell ^[deptypes=build] diffutils@=3.9%gcc@=11.2.0 build_system=autotools arch=cray-cnl7-haswell ^[deptypes=link] gcc-runtime@=11.2.0%gcc@=11.2.0 build_system=generic arch=cray-cnl7-haswell ^[deptypes=build,link] gdbm@=1.23%gcc@=11.2.0 build_system=autotools arch=cray-cnl7-haswell ^[deptypes=build] gmake@=4.4.1%gcc@=11.2.0~guile build_system=generic arch=cray-cnl7-haswell ^[deptypes=build,link virtuals=iconv] libiconv@=1.17%gcc@=11.2.0 build_system=autotools libs=shared,static arch=cray-cnl7-haswell ^[deptypes=build,link] ncurses@=6.4%gcc@=11.2.0~symlinks+termlib abi=none build_system=autotools arch=cray-cnl7-haswell ^[deptypes=build,link] nghttp2@=1.57.0%gcc@=11.2.0 build_system=autotools arch=cray-cnl7-haswell ^[deptypes=build,link] openssl@=3.1.3%gcc@=11.2.0~docs+shared build_system=generic certs=mozilla arch=cray-cnl7-haswell ^[deptypes=build] perl@=5.38.0%gcc@=11.2.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=cray-cnl7-haswell ^[deptypes=build virtuals=pkgconfig] pkgconf@=1.9.5%gcc@=11.2.0 build_system=autotools arch=cray-cnl7-haswell ^[deptypes=build,link] readline@=8.2%gcc@=11.2.0 build_system=autotools patches=bbf97f1 arch=cray-cnl7-haswell ^[deptypes=build,run] unzip@=6.0%gcc@=11.2.0 build_system=makefile arch=cray-cnl7-haswell ^[deptypes=build,link virtuals=zlib-api] zlib-ng@=2.1.5%gcc@=11.2.0+compat+opt build_system=autotools arch=cray-cnl7-haswell
```

It sounds like we are facing a spack limitation.
It seems that `neovim` depends on two Lua providers (I have to check what are the guidelines about this)

https://github.com/spack/spack/blob/b70cb60b6581ee4b02f786de516aa6a24457c063/var/spack/repos/builtin/packages/neovim/package.py#L87-L89

and then dependencies like `lua-bitlib` are enabled when a specific `lua-lang` provider is detected, i.e. `lua`.

https://github.com/spack/spack/blob/b70cb60b6581ee4b02f786de516aa6a24457c063/var/spack/repos/builtin/packages/neovim/package.py#L92-L93

I don't know anything about Lua and its packaging system. Just pin-pointing things that might ring some bell to who knows more.
I think that the problem relies in the `neovim` package itself. From a quick investigation, these two libraries should be enabled just when a non-jit `Lua` is used.

It seems that

https://github.com/spack/spack/blob/b70cb60b6581ee4b02f786de516aa6a24457c063/var/spack/repos/builtin/packages/neovim/package.py#L92-L93

induces the concretizer to end up with two different Lua implementations in the DAG, probably due to 

https://github.com/spack/spack/blob/b70cb60b6581ee4b02f786de516aa6a24457c063/var/spack/repos/builtin/packages/neovim/package.py#L87-L88

triggering the assertion.

I'm going to open a draft PR to give it a try.
> I was able to reproduce it with a minimal recipe, it seems to come from confusions about the lua-lang+luajit double-virtual resolution:
> 
> ```python
> from spack.package import *
> class Testlua(BundlePackage):
>     homepage = ""
>     version("1")
>     depends_on("lua-lang")
>     depends_on("luajit")
>     depends_on("lua-bitlib")
> ```

This is probably not a compatible set, the bitlib is a reimplementation of built-in functionality of luajit, they shouldn't get combined.  

The valid combinations are:
* ^lua-lang=luajit
* ^lua-lang=lua, lua-ffi, lua-bitlib

Maybe we need an explicit conflict to ensure other combinations don't happen?

Resolving patch:
diff --git a/lib/spack/spack/build_systems/lua.py b/lib/spack/spack/build_systems/lua.py
--- a/lib/spack/spack/build_systems/lua.py
+++ b/lib/spack/spack/build_systems/lua.py
@@ -29,15 +29,12 @@ class LuaPackage(spack.package_base.PackageBase):
 
     with when("build_system=lua"):
         depends_on("lua-lang")
-        extends("lua", when="^lua")
-        with when("^lua-luajit"):
-            extends("lua-luajit")
-            depends_on("luajit")
-            depends_on("lua-luajit+lualinks")
-        with when("^lua-luajit-openresty"):
-            extends("lua-luajit-openresty")
-            depends_on("luajit")
-            depends_on("lua-luajit-openresty+lualinks")
+        with when("^[virtuals=lua-lang] lua"):
+            extends("lua")
+        with when("^[virtuals=lua-lang] lua-luajit"):
+            extends("lua-luajit+lualinks")
+        with when("^[virtuals=lua-lang] lua-luajit-openresty"):
+            extends("lua-luajit-openresty+lualinks")
 
     @property
     def lua(self):


Test patch: