Repository: spack/spack

Issue:
Weird dependency resolution error
This command

```
spack install -j1 -v cactusext +funhpc +simulationio %gcc@6.1.0-spack
```

leads to this error from Spack:

```
==> Error: Multiple providers found for 'mpi': ['openmpi%gcc@6.1.0-spack~pmi+verbs', 'openmpi@1.10.2%gcc@6.1.0-spack~pmi~psm~tm+verbs=linux-x86_64^hwloc@1.11.3%gcc@6.1.0-spack=linux-x86_64^libpciaccess@0.13.4%gcc@6.1.0-spack=linux-x86_64^libsigsegv@2.10%gcc@6.1.0-spack=linux-x86_64^libtool@2.4.6%gcc@6.1.0-spack=linux-x86_64^m4@1.4.17%gcc@6.1.0-spack+sigsegv=linux-x86_64']
```

After adding some line breaks, this becomes

```
openmpi%gcc@6.1.0-spack~pmi+verbs

openmpi@1.10.2%gcc@6.1.0-spack~pmi~psm~tm+verbs=linux-x86_64
    ^hwloc@1.11.3%gcc@6.1.0-spack=linux-x86_64
    ^libpciaccess@0.13.4%gcc@6.1.0-spack=linux-x86_64
    ^libsigsegv@2.10%gcc@6.1.0-spack=linux-x86_64
    ^libtool@2.4.6%gcc@6.1.0-spack=linux-x86_64
    ^m4@1.4.17%gcc@6.1.0-spack+sigsegv=linux-x86_64
```

I see that the first (essentially `openmpi +verbs`) is what I specify, and the second is what one obtains naturally after adding all dependencies, versions, and architectures. I don't see where the conflict comes from.

How do I debug this? I realize this involves some packages of mine that are only available in my branch of Spack (see https://github.com/eschnett/spack/tree/eschnett/cactusext). Is there a particular routine to look at, or some debug output that would trace Spack's dependency resolution mechanism?



Discussion:
I believe this is my fault. The first OpenMPI installation was probably done during the short time in which I removed OpenMPI's dependency on hwloc. The second installation was from before or after.

Oooh. Good idea; I'll remove the existing install. I thought that existing packages didn't matter.

... no, this didn't help:

```
$ spack find openmpi
==> 1 installed packages.
-- linux-x86_64 / gcc@5.3.0-spack -------------------------------
openmpi@1.10.2~psm~tm~verbs
```

(This older install uses a different compiler.)

```
$ spack install $installopts cactusext +funhpc +simulationio %gcc@6.1.0-spack
==> Error: Multiple providers found for 'mpi': ['openmpi%gcc@6.1.0-spack~pmi+verbs', 'openmpi@1.10.2%gcc@6.1.0-spack~pmi~psm~tm+verbs=linux-x86_64^hwloc@1.11.3%gcc@6.1.0-spack=linux-x86_64^libpciaccess@0.13.4%gcc@6.1.0-spack=linux-x86_64^libsigsegv@2.10%gcc@6.1.0-spack=linux-x86_64^libtool@2.4.6%gcc@6.1.0-spack=linux-x86_64^m4@1.4.17%gcc@6.1.0-spack+sigsegv=linux-x86_64']
eschnetter@compute2:/xfs1/eschnetter/src/spack (13:02:41)
```

This is odd because those two specs should be combined in concretization.  Is there a fork I can look at that has your packages?

Yes; see https://github.com/eschnett/spack/tree/eschnett/cactusext.

I now see that it works on two other machines. Maybe it's something system-specific, or my Spack install is broken. I'm going to delete all packages and try again.

I would definitely delete all OpenMPI installations at the least. Hopefully that solves everything.

The problem persists.

I've encountered a second system where I see this error. Can someone help me debug this? Is there a particular routine where I could insert debug output? Any pointers would be appreciated.

Hmm, when you uninstalled all MPI packages, did you delete the directories manually and run `spack reindex`, or did you just `spack uninstall` everything?

What's the output of `spack config get compilers`?

> I realize this involves some packages of mine that are only available in my branch of Spack

Does your branch involve changes to the OpenMPI packages?

> openmpi%gcc@6.1.0-spack~pmi+verbs
> 
> openmpi@1.10.2%gcc@6.1.0-spack~pmi~psm~tm+verbs=linux-x86_64
>     ^hwloc@1.11.3%gcc@6.1.0-spack=linux-x86_64
>     ^libpciaccess@0.13.4%gcc@6.1.0-spack=linux-x86_64
>     ^libsigsegv@2.10%gcc@6.1.0-spack=linux-x86_64
>     ^libtool@2.4.6%gcc@6.1.0-spack=linux-x86_64
>     ^m4@1.4.17%gcc@6.1.0-spack+sigsegv=linux-x86_64

Not only does the second OpenMPI installation include more dependencies. It also includes different variants. The problem has to be that one version was installed when OpenMPI had no dependencies and had no psm/tm variants, while the other version did have a hwloc dependency and psm/tm variants.

Actually, just looked at your `cactusext` package. It is riddled with bugs. First of all, it depends on OpenMPI 5 times!:

``` python
depends_on("mpi")
depends_on("openmpi")
depends_on("mpi %"+cactusext_compiler)
depends_on("openmpi %"+cactusext_compiler)
depends_on("openmpi"+''.join([" "+opt for opt in openmpi_opts]))
```

The first line of that list would suffice. If you want to specify which MPI implementation or compiler to use, just run:

```
spack install cactustext %gcc@6.1.0-spack ^openmpi+verbs
```

Spack will take care of the rest, like deciding which MPI package to link to and which compiler each dependency should be built with. If you would like to open a PR, I would be more than happy to help you clean it up.

Is it indeed a bug to depend on the same package multiple times? I assumed that Spack would automatically combine these dependencies. If not, I can trivially rewrite this.

I don't want to have to explicitly specify `^openmpi+verbs` on the command line, since this quickly becomes tedious. For example, I want to say "build with verbs if verbs is supported on the system; otherwise, build without verbs". The respective logic is not in OpenMPI (where it should be), and also not in Spack's OpenMPI package (I made a respective suggestion, but didn't receive any feedback). So I'm putting this check into a package of my own, `cactusext`, where I can test it.

However, I need not just `openmpi` -- I need quite a long list of packages, each with its own settings, and specifying all the dependencies and compilers and settings on the command line is very tedious. For example, this is what I need for `petsc` on Darwin:

```
spack install $installopts petsc %gcc@5.3.0-spack +boost +hdf5 +mpi ^boost+mpi ^hdf5+mpi ^openblas ^openmpi ^python %clang@7.3.0-apple ^bzip2%gcc@5.3.0-spack ^ncurses%gcc@5.3.0-spack ^readline%gcc@5.3.0-spack ^openssl%gcc@5.3.0-spack ^sqlite%gcc@5.3.0-spack ^zlib%gcc@5.3.0-spack
```

since PETSc requires Python, and Python needs the system compiler on Darwin, but I want a C++14-capable compiler for other packages. (Essentially, I want to only build Python with the system compiler, and everything else with GCC, and these are the options that make Spack do this.) There are similar issues with `jemalloc` and a few other packages.

I used to specify all this manually, and then thought about writing a script that does so, and then realized that I should just write a (personal) Spack package for this. Hence there are many conditions in `cactusext`, working around some current shortcomings in Spack and some of the packages, and making some very specific choices.

> Is it indeed a bug to depend on the same package multiple times? I assumed that Spack would automatically combine these dependencies.

I don't think Spack is currently smart enough to do this. Try reducing it to just one MPI dependence and see if that solves the problem.

@adamjstewart Thanks for the pointer. Unfortunately, this doesn't help either. I now have these dependencies

```
blas ^openblas 
boost +mpi 
fftw +mpi +openmp %gcc@6.1.0-spack 
funhpc %gcc@6.1.0-spack when=+funhpc
git %clang@7.3.0-apple 
gsl %gcc@6.1.0-spack 
hdf5 +mpi @1.10.0 %gcc@6.1.0-spack 
hwloc %gcc@6.1.0-spack 
jemalloc %clang@7.3.0-apple when=+funhpc
julia %gcc@6.1.0-spack when=+julia
lapack ^openblas 
libsigsegv  
llvm %gcc@6.1.0-spack when=+llvm
lua %gcc@6.1.0-spack 
mpi ^openmpi 
openblas %gcc@6.1.0-spack 
openmpi %gcc@6.1.0-spack 
openssl %gcc@6.1.0-spack 
papi %gcc@6.1.0-spack 
petsc +boost +hdf5 +mpi %gcc@6.1.0-spack ~mumps 
python @2.7.11 %clang@7.3.0-apple 
simulationio %gcc@6.1.0-spack when=+simulationio
zlib %gcc@6.1.0-spack 
```

which still lead to the same error

```
==> Error: Multiple providers found for 'mpi': ['openmpi%gcc@6.1.0-spack', 'openmpi@1.10.2%gcc@6.1.0-spack~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs=darwin-x86_64^hwloc@1.11.3%gcc@6.1.0-spack=darwin-x86_64^libpciaccess@0.13.4%gcc@6.1.0-spack=darwin-x86_64^libsigsegv@2.10%gcc@6.1.0-spack=darwin-x86_64^libtool@2.4.6%gcc@6.1.0-spack=darwin-x86_64^m4@1.4.17%gcc@6.1.0-spack+sigsegv=darwin-x86_64']
```

Is that error message from running `spack spec cactusext` or `spack install cactusext`? Does it include a line number that it is occurring on?

That is the error from `spack install`. `spack spec` seems to return the same error. There is no line number, but when I replace the `raise` by `assert`, I see that it is in the routine `_find_provider` as called from the last line in this snippet from `spec.py`:

```
        # If it's a virtual dependency, try to find an existing
        # provider in the spec, and merge that.
        if dep.virtual:
            visited.add(dep.name)
            provider = self._find_provider(dep, provider_index)
```

This is the assert backtrace -- the line numbers are now off because I started inserting `print` statements. So far, I'm still using shotgun debugging without much understanding of what is going on.
``
Traceback (most recent call last):
  File "/Users/eschnett/src/spack/bin/spack", line 176, in <module>
    main()
  File "/Users/eschnett/src/spack/bin/spack", line 154, in main
    return_val = command(parser, args)
  File "/Users/eschnett/src/spack/lib/spack/spack/cmd/spec.py", line 58, in spec
    spec.concretize()
  File "/Users/eschnett/src/spack/lib/spack/spack/spec.py", line 949, in concretize
    changes = (self.normalize(force),
  File "/Users/eschnett/src/spack/lib/spack/spack/spec.py", line 1234, in normalize
    any_change = self._normalize_helper(visited, spec_deps, provider_index)
  File "/Users/eschnett/src/spack/lib/spack/spack/spec.py", line 1194, in _normalize_helper
    pkg_dep, visited, spec_deps, provider_index)
  File "/Users/eschnett/src/spack/lib/spack/spack/spec.py", line 1164, in _merge_dependency
    changed |= dependency._normalize_helper(visited, spec_deps, provider_index)
  File "/Users/eschnett/src/spack/lib/spack/spack/spec.py", line 1194, in _normalize_helper
    pkg_dep, visited, spec_deps, provider_index)
  File "/Users/eschnett/src/spack/lib/spack/spack/spec.py", line 1125, in _merge_dependency
    provider = self._find_provider(dep, provider_index)
  File "/Users/eschnett/src/spack/lib/spack/spack/spec.py", line 1087, in _find_provider
    assert False
AssertionError

```
```

Here is a bit more information. The original dependencies are:

```
blas ^openblas 
boost +mpi 
fftw +mpi +openmp %gcc@6.1.0-spack 
funhpc %gcc@6.1.0-spack when=+funhpc
git %clang@7.3.0-apple ^autoconf %gcc@6.1.0-spack ^curl %gcc@6.1.0-spack ^expat %gcc@6.1.0-spack ^openssl %gcc@6.1.0-spack ^zlib %gcc@6.1.0-spack 
gsl %gcc@6.1.0-spack 
hdf5 +mpi @1.10.0 %gcc@6.1.0-spack 
hwloc %gcc@6.1.0-spack 
jemalloc %clang@7.3.0-apple when=+funhpc
julia %gcc@6.1.0-spack when=+julia
lapack ^openblas 
llvm %gcc@6.1.0-spack when=+llvm
lua %gcc@6.1.0-spack 
mpi ^openmpi 
openblas %gcc@6.1.0-spack 
openmpi %gcc@6.1.0-spack 
openssl %gcc@6.1.0-spack 
papi %gcc@6.1.0-spack 
petsc +boost +hdf5 +mpi %gcc@6.1.0-spack ~mumps 
python @2.7.11 %clang@7.3.0-apple ^bzip2 %gcc@6.1.0-spack ^ncurses %gcc@6.1.0-spack ^readline %gcc@6.1.0-spack ^openssl %gcc@6.1.0-spack ^sqlite %gcc@6.1.0-spack ^zlib %gcc@6.1.0-spack 
simulationio %gcc@6.1.0-spack when=+simulationio
zlib %gcc@6.1.0-spack 
```

`spack spec` then outputs:

```
  cactusext%gcc@6.1.0-spack+simulationio
      ^boost+mpi
          ^openmpi%gcc@6.1.0-spack
              ^hwloc%gcc@6.1.0-spack
                  ^libpciaccess
                      ^libtool
                          ^m4
      ^fftw%gcc@6.1.0-spack+mpi+openmp
      ^git%clang@7.3.0-apple
          ^autoconf
          ^curl
              ^openssl%gcc@6.1.0-spack
                  ^zlib%gcc@6.1.0-spack
          ^expat
              ^cmake
      ^gsl%gcc@6.1.0-spack
      ^hdf5@1.10.0%gcc@6.1.0-spack+mpi
      ^lua%gcc@6.1.0-spack
          ^ncurses
          ^readline
      ^openblas%gcc@6.1.0-spack
      ^papi%gcc@6.1.0-spack
      ^petsc%gcc@6.1.0-spack+boost+hdf5+mpi~mumps
          ^openblas%gcc@6.1.0-spack
          ^python@2.7.11%clang@7.3.0-apple
              ^bzip2
              ^sqlite
      ^simulationio%gcc@6.1.0-spack
          ^swig
              ^pcre
```

and then crashes during concretization. However, I believe this output is already wrong, as it is missing the `openmpi` dependency of `cactusext`. It is still there as indirect dependency, but not any more as direct dependency.

What happens when you remove the openmpi dependency altogether?

@eschnett Are you building your own openmpi, or is the dependency supposed to be satisfied by an external package? If the latter, could you post the relevant piece of your packages.yaml file?

Also there is no need to have 5 separate dependencies on mpi to get the behavior you want. Those can be combined into 

```
cond_plus = '+' if openmpi_opts else ""
depends_on('openmpi %' + cactusext_compiler + cond_plus + '+'.join(opt for opt in openmpi_opts))
```

the dependency on mpi separately from openmpi is superceded by the openmpi dependency, and all of the other conditions can be thrown onto one line.

And just to make sure I'm on the same page before I look into this, you said that `spack spec` has been normalizing properly, but that it fails when concretization calls normalize?

-Greg

I've modified (but not yet pushed) a change that creates one dependency line only for each dependency. This did not change anything.

Regarding `spack spec`: I think that the normalization is already wrong, as I have an explicit dependency on `openmpi` that is missing in the `spack spec` output. You see above that I have `openmpi` as explicit dependency (with a compiler specifier), yet the output below does not list `openmpi`.

Can you push that change? It will be substantially easier to reason about when I'm trying to debug this.

Done. Thanks!

I'm building Spack's standard OpenMPI package. I don't think I have any changes compared to the `development` branch.

I’ve found one bug so far, but I don’t believe it solves the issue.

The syntax `depends_on(‘mpi ^openmpi’) is not legal. MPI does not depend on openmpi, it is provided by openmpi.

To check this, try `spack spec mpi ^openmpi`. This will raise an error.

-Greg

From: Erik Schnetter <notifications@github.com<mailto:notifications@github.com>>
Reply-To: LLNL/spack <reply@reply.github.com<mailto:reply@reply.github.com>>
Date: Monday, May 16, 2016 at 5:00 PM
To: LLNL/spack <spack@noreply.github.com<mailto:spack@noreply.github.com>>
Cc: Gregory Becker <becker33@llnl.gov<mailto:becker33@llnl.gov>>, Comment <comment@noreply.github.com<mailto:comment@noreply.github.com>>
Subject: Re: [LLNL/spack] Weird dependency resolution error (#928)

I'm building Spack's standard OpenMPI package. I don't think I have any changes compared to the development branch.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHubhttps://github.com/LLNL/spack/issues/928#issuecomment-219583492

I added these dependencies while playing around; I also tried without. Thanks for the pointer -- I removed them again.

@eschnett It appears that spack is incorrectly raising a MultipleProvidersError when it encounters multiple providers for a virtual dependency that could be concretized to the same value.

I don't want to push directly to your repo, and the changes are tiny, so I will list them here. Try making the following code changes to the _find_provider() method on the Spec class in lib/spack/spack/spec.py. There are 5 lines to add, marked with the `>` symbol.

```
    if providers:
>       # Remove duplicate providers that can concretize to the same result.
>       for provider in providers:
>           for spec in providers:
>               if spec is not provider and spec.satisfies(provider):
>                   providers.remove(spec)
        # Can't have multiple providers for the same thing in one spec.
        if len(providers) > 1:
            raise MultipleProviderError(vdep, providers)
            return providers[0]
```

Can you confirm that this works for all of your needs and submit a PR to the mainline if so?

@eschnett I just made one small edit to the code, please recheck it if you had already looked at it.

Yay! Success.

Might want to put this in a PR.


Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1071,6 +1071,11 @@ def _find_provider(self, vdep, provider_index):
         # If there is a provider for the vpkg, then use that instead of
         # the virtual package.
         if providers:
+            # Remove duplicate providers that can concretize to the same result.
+            for provider in providers:
+                for spec in providers:
+                    if spec is not provider and provider.satisfies(spec):
+                        providers.remove(spec)
             # Can't have multiple providers for the same thing in one spec.
             if len(providers) > 1:
                 raise MultipleProviderError(vdep, providers)


Test patch: