Repository: spack/spack

Issue:
ValueError: Specs passed to a DirectoryLayout must be concrete!
I've never used modules before, but ANL's LCRC is looking to make the switch to Lmod for our next cluster. I decided to try out our modules tutorial, but ran into a problem. I created a `~/.spack/modules.yaml` containing:
```yaml
modules:
  enable::
    - lmod
```
and ran:
```
$ spack module refresh --module-type lmod
```
but it crashed with:
```
Traceback (most recent call last):
  File "/soft/spack-0.10.0/bin/spack", line 212, in <module>
    main(sys.argv)
  File "/soft/spack-0.10.0/bin/spack", line 208, in main
    _main(args, unknown)
  File "/soft/spack-0.10.0/bin/spack", line 174, in _main
    return_val = command(parser, args)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/cmd/module.py", line 259, in module
    callbacks[args.subparser_name](module_type, specs, args)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/cmd/module.py", line 243, in refresh
    x.write(overwrite=True)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/modules.py", line 380, in write
    set_module_variables_for_package(package, mod)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/build_environment.py", line 380, in set_module_variables_for_package
    m.std_cmake_args = spack.CMakePackage._std_args(pkg)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/build_systems/cmake.py", line 95, in _std_args
    rpaths = ':'.join(spack.build_environment.get_rpaths(pkg))
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/build_environment.py", line 424, in get_rpaths
    rpaths.extend(d.prefix.lib for d in deps
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/build_environment.py", line 425, in <genexpr>
    if os.path.isdir(d.prefix.lib))
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/spec.py", line 1156, in prefix
    return Prefix(spack.store.layout.path_for_spec(self))
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/directory_layout.py", line 124, in path_for_spec
    _check_concrete(spec)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/directory_layout.py", line 42, in _check_concrete
    raise ValueError('Specs passed to a DirectoryLayout must be concrete!')
ValueError: Specs passed to a DirectoryLayout must be concrete!
```
This seems to be a common error message. I'm seeing it in #206, #217, #238, #244, and #1960. From what I gather from reading those other issues, it sounds like this is caused by changing hashes and packages/variants being added/removed? Unfortunately, this makes Spack + Modules a pipe dream, as you would have to completely reinstall Spack or uninstall all previous versions the first time any conflict occurs.
ValueError: Specs passed to a DirectoryLayout must be concrete!
I've never used modules before, but ANL's LCRC is looking to make the switch to Lmod for our next cluster. I decided to try out our modules tutorial, but ran into a problem. I created a `~/.spack/modules.yaml` containing:
```yaml
modules:
  enable::
    - lmod
```
and ran:
```
$ spack module refresh --module-type lmod
```
but it crashed with:
```
Traceback (most recent call last):
  File "/soft/spack-0.10.0/bin/spack", line 212, in <module>
    main(sys.argv)
  File "/soft/spack-0.10.0/bin/spack", line 208, in main
    _main(args, unknown)
  File "/soft/spack-0.10.0/bin/spack", line 174, in _main
    return_val = command(parser, args)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/cmd/module.py", line 259, in module
    callbacks[args.subparser_name](module_type, specs, args)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/cmd/module.py", line 243, in refresh
    x.write(overwrite=True)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/modules.py", line 380, in write
    set_module_variables_for_package(package, mod)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/build_environment.py", line 380, in set_module_variables_for_package
    m.std_cmake_args = spack.CMakePackage._std_args(pkg)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/build_systems/cmake.py", line 95, in _std_args
    rpaths = ':'.join(spack.build_environment.get_rpaths(pkg))
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/build_environment.py", line 424, in get_rpaths
    rpaths.extend(d.prefix.lib for d in deps
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/build_environment.py", line 425, in <genexpr>
    if os.path.isdir(d.prefix.lib))
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/spec.py", line 1156, in prefix
    return Prefix(spack.store.layout.path_for_spec(self))
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/directory_layout.py", line 124, in path_for_spec
    _check_concrete(spec)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/directory_layout.py", line 42, in _check_concrete
    raise ValueError('Specs passed to a DirectoryLayout must be concrete!')
ValueError: Specs passed to a DirectoryLayout must be concrete!
```
This seems to be a common error message. I'm seeing it in #206, #217, #238, #244, and #1960. From what I gather from reading those other issues, it sounds like this is caused by changing hashes and packages/variants being added/removed? Unfortunately, this makes Spack + Modules a pipe dream, as you would have to completely reinstall Spack or uninstall all previous versions the first time any conflict occurs.


Discussion:
> Unfortunately, this makes Spack + Modules a pipe dream, as you would have to completely reinstall Spack or uninstall all previous versions the first time any conflict occurs.

@adamjstewart Does this happen if you use a fresh checkout of `spack`? 
> Does this happen if you use a fresh checkout of spack?

Nope. I'm assuming that this is because none of the hashes have changed since I checked it out and installed a few packages.
@adamjstewart I think you are right. In my opinion Spack should work consistently *at a given version*. Constantly updating a checkout of `develop` is very likely to be causing these kind of problems and I am not really sure we want to support this use case. I understand this was an hard decision to make before because the releases were not so frequent, but apparently this is going to change. @tgamblin ?
@adamjstewart Anyhow, I took the issue. I'll try to see as soon as I'll have time (sorry, busy week) if I can find a way to reproduce it.
Steps to reproduce this error from a fresh clone of Spack:

1. `spack install m4`
2. `spack edit libsigsegv` and add `variant('fake')`
3. `spack install m4`
4. `spack module refresh`
> In my opinion Spack should work consistently at a given version. Constantly updating a checkout of develop is very likely to be causing these kind of problems and I am not really sure we want to support this use case. I understand this was an hard decision to make before because the releases were not so frequent, but apparently this is going to change.

I'm not sure that this is realistic. What you are telling me is that if a user asks me for a new package, or asks me to add some additional support to a package, I tell them "sure, but you'll have to wait until the next Spack release before I can install it for you".
The problem here seems to be that with the new variant added for `libsigsegv` the previously installed `m4` is not seen as concrete (very likely because there's no value for `fake` in the previous `libsigsev`).
@adamjstewart I'll try my best to have a look at this tomorrow :smile: 
@adamjstewart: I think we should do our best to work across versions.
Thanks @alalazo! I've talked to @tgamblin before about the changing hashes problem and he assured me that I shouldn't have to reinstall Spack every time a variant is added or the hashing algorithm changes. I might end up with hundreds of duplicate installations (currently pushing 1,500 across 3 Spack installations), but that they shouldn't cause any problems. So I'm hoping this is just a "bug" and not a "feature".
@alalazo:
> The problem here seems to be that with the new variant added for libsigsegv the previously installed m4 is not seen as concrete (very likely because there's no value for fake in the previous libsigsev).

Specs of already-installed packages are explicitly read in as concrete. Concrete specs cache both the fact that they are concrete (in `self._concrete`) and the hash they were created with (in `self._hash`) specifically so that we respect the hash and package configuration *from the time the package was built*.  This addresses package changes over time: if a spec is marked concrete, you know it has values for *all* variants *that existed when it was built*.  A missing variant on a concrete spec means the package didn't know about that variant, and a spec without variant `foo` won't satisfy *either* `+foo` or `~foo`. 

So basically I don't think it's because Spack is failing to recognize that a previously installed spec is concrete.  If it is, it's a bug.




@adamjstewart: 
> From what I gather from reading those other issues, it sounds like this is caused by changing hashes and packages/variants being added/removed?

Spack has a `DirectoryLayout` class that controls how packages are laid out under the install root (usually `opt/spack`).  Among other things, the directory layout guarantees you a unique, deterministically knowable directory for every installed package.  This error is raised when you ask it for a directory corresponding to a spec that is not concrete -- i.e. a spec that could represent more than one unique configuration of a package.

In 2015, Spack didn't do a very good job of managing changing package configurations.  If you added a new variant or a new dependency to a package, Spack would not recognize that old versions of a package were concrete, because the new DAG was a superset of the old one.  The PRs you linked to fixed that issue -- now we remember when an old build was concrete and new builds will have different hashes from old ones, but you can still compare them to each other without getting issues.
@alalazo: does the module infrastructure try to rebuild or reconcretize old specs rather than sticking with the ones read in from the DB?  If it does, then you can run into this problem.
> does the module infrastructure try to rebuild or reconcretize old specs rather than sticking with the ones read in from the DB?

 I am checking that right now.
Possibly useful debugging information.

#### Things that do not cause a problem

```
$ spack install libsigsegv
$ spack edit libsigsegv  # add variant
$ spack install libsigsegv
$ spack module refresh
```
```
$ spack install libsigsegv
$ spack install m4
$ spack edit libsigsegv  # add variant
$ spack install libsigsegv
$ spack install m4
$ spack module refresh
```

#### Things that do cause a problem
```
$ spack install m4
$ spack edit libsigsegv  # add variant
$ spack install m4
$ spack module refresh
```
It seems to make a difference whether or not the package was explicitly installed.
@tgamblin I am investigating where this is coming from:
```python
In[13]: print pkg.spec
m4@1.4.18%gcc@4.8+sigsegv arch=linux-ubuntu14-x86_64 ^libsigsegv@2.10%gcc@4.8 arch=linux-ubuntu14-x86_64
In[14]: pkg.spec.concrete
Out[15]: True
In[15]: pkg.spec['libsigsegv'].concrete
Out[16]: False
```
Debugging objects right before the exception is raised. The `m4` package is clearly inconsistent there.
> So basically I don't think it's because Spack is failing to recognize that a previously installed spec is concrete. If it is, it's a bug.

@tgamblin I think it's a bug:
```python
# a is m4@1.4.18%gcc@4.8+sigsegv arch=linux-ubuntu14-x86_64 ^libsigsegv@2.10%gcc@4.8 arch=linux-ubuntu14-x86_64
a = specs[1]
a.concrete, a['libsigsegv'].concrete
Out[3]: (True, True)
a = specs[1].package.spec
a.concrete, a['libsigsegv'].concrete
Out[5]: (True, False)
```
In the scope of the main `module` command, before even entering the refresh subcommand. It seems that `spec is not spec.package.spec`.
Ok, the problem seems to be inherent in spec construction from records in the DB. If there's no rush I'll try to put together a PR tomorrow.
Going with a debugger through the failing case provided by @adamjstewart I see that `spec.package` is **always** read from the current repository of packages. This seems to be inconsistent to me. As we store the `package.py` files used to build a particular DAG in the `.spack` subfolder of the prefix I think we should read the packages from there if we retrieve something from a DB. Do you agree @tgamblin or am I overlooking something?
> Unfortunately, this makes Spack + Modules a pipe dream, as you would have to completely reinstall Spack or uninstall all previous versions the first time any conflict occurs.

@adamjstewart Does this happen if you use a fresh checkout of `spack`? 
> Does this happen if you use a fresh checkout of spack?

Nope. I'm assuming that this is because none of the hashes have changed since I checked it out and installed a few packages.
@adamjstewart I think you are right. In my opinion Spack should work consistently *at a given version*. Constantly updating a checkout of `develop` is very likely to be causing these kind of problems and I am not really sure we want to support this use case. I understand this was an hard decision to make before because the releases were not so frequent, but apparently this is going to change. @tgamblin ?
@adamjstewart Anyhow, I took the issue. I'll try to see as soon as I'll have time (sorry, busy week) if I can find a way to reproduce it.
Steps to reproduce this error from a fresh clone of Spack:

1. `spack install m4`
2. `spack edit libsigsegv` and add `variant('fake')`
3. `spack install m4`
4. `spack module refresh`
> In my opinion Spack should work consistently at a given version. Constantly updating a checkout of develop is very likely to be causing these kind of problems and I am not really sure we want to support this use case. I understand this was an hard decision to make before because the releases were not so frequent, but apparently this is going to change.

I'm not sure that this is realistic. What you are telling me is that if a user asks me for a new package, or asks me to add some additional support to a package, I tell them "sure, but you'll have to wait until the next Spack release before I can install it for you".
The problem here seems to be that with the new variant added for `libsigsegv` the previously installed `m4` is not seen as concrete (very likely because there's no value for `fake` in the previous `libsigsev`).
@adamjstewart I'll try my best to have a look at this tomorrow :smile: 
@adamjstewart: I think we should do our best to work across versions.
Thanks @alalazo! I've talked to @tgamblin before about the changing hashes problem and he assured me that I shouldn't have to reinstall Spack every time a variant is added or the hashing algorithm changes. I might end up with hundreds of duplicate installations (currently pushing 1,500 across 3 Spack installations), but that they shouldn't cause any problems. So I'm hoping this is just a "bug" and not a "feature".
@alalazo:
> The problem here seems to be that with the new variant added for libsigsegv the previously installed m4 is not seen as concrete (very likely because there's no value for fake in the previous libsigsev).

Specs of already-installed packages are explicitly read in as concrete. Concrete specs cache both the fact that they are concrete (in `self._concrete`) and the hash they were created with (in `self._hash`) specifically so that we respect the hash and package configuration *from the time the package was built*.  This addresses package changes over time: if a spec is marked concrete, you know it has values for *all* variants *that existed when it was built*.  A missing variant on a concrete spec means the package didn't know about that variant, and a spec without variant `foo` won't satisfy *either* `+foo` or `~foo`. 

So basically I don't think it's because Spack is failing to recognize that a previously installed spec is concrete.  If it is, it's a bug.




@adamjstewart: 
> From what I gather from reading those other issues, it sounds like this is caused by changing hashes and packages/variants being added/removed?

Spack has a `DirectoryLayout` class that controls how packages are laid out under the install root (usually `opt/spack`).  Among other things, the directory layout guarantees you a unique, deterministically knowable directory for every installed package.  This error is raised when you ask it for a directory corresponding to a spec that is not concrete -- i.e. a spec that could represent more than one unique configuration of a package.

In 2015, Spack didn't do a very good job of managing changing package configurations.  If you added a new variant or a new dependency to a package, Spack would not recognize that old versions of a package were concrete, because the new DAG was a superset of the old one.  The PRs you linked to fixed that issue -- now we remember when an old build was concrete and new builds will have different hashes from old ones, but you can still compare them to each other without getting issues.
@alalazo: does the module infrastructure try to rebuild or reconcretize old specs rather than sticking with the ones read in from the DB?  If it does, then you can run into this problem.
> does the module infrastructure try to rebuild or reconcretize old specs rather than sticking with the ones read in from the DB?

 I am checking that right now.
Possibly useful debugging information.

#### Things that do not cause a problem

```
$ spack install libsigsegv
$ spack edit libsigsegv  # add variant
$ spack install libsigsegv
$ spack module refresh
```
```
$ spack install libsigsegv
$ spack install m4
$ spack edit libsigsegv  # add variant
$ spack install libsigsegv
$ spack install m4
$ spack module refresh
```

#### Things that do cause a problem
```
$ spack install m4
$ spack edit libsigsegv  # add variant
$ spack install m4
$ spack module refresh
```
It seems to make a difference whether or not the package was explicitly installed.
@tgamblin I am investigating where this is coming from:
```python
In[13]: print pkg.spec
m4@1.4.18%gcc@4.8+sigsegv arch=linux-ubuntu14-x86_64 ^libsigsegv@2.10%gcc@4.8 arch=linux-ubuntu14-x86_64
In[14]: pkg.spec.concrete
Out[15]: True
In[15]: pkg.spec['libsigsegv'].concrete
Out[16]: False
```
Debugging objects right before the exception is raised. The `m4` package is clearly inconsistent there.
> So basically I don't think it's because Spack is failing to recognize that a previously installed spec is concrete. If it is, it's a bug.

@tgamblin I think it's a bug:
```python
# a is m4@1.4.18%gcc@4.8+sigsegv arch=linux-ubuntu14-x86_64 ^libsigsegv@2.10%gcc@4.8 arch=linux-ubuntu14-x86_64
a = specs[1]
a.concrete, a['libsigsegv'].concrete
Out[3]: (True, True)
a = specs[1].package.spec
a.concrete, a['libsigsegv'].concrete
Out[5]: (True, False)
```
In the scope of the main `module` command, before even entering the refresh subcommand. It seems that `spec is not spec.package.spec`.
Ok, the problem seems to be inherent in spec construction from records in the DB. If there's no rush I'll try to put together a PR tomorrow.
Going with a debugger through the failing case provided by @adamjstewart I see that `spec.package` is **always** read from the current repository of packages. This seems to be inconsistent to me. As we store the `package.py` files used to build a particular DAG in the `.spack` subfolder of the prefix I think we should read the packages from there if we retrieve something from a DB. Do you agree @tgamblin or am I overlooking something?

Resolving patch:
diff --git a/lib/spack/spack/database.py b/lib/spack/spack/database.py
--- a/lib/spack/spack/database.py
+++ b/lib/spack/spack/database.py
@@ -340,6 +340,7 @@ def invalid_record(hash_key, error):
         # cached prematurely.
         for hash_key, rec in data.items():
             rec.spec._mark_concrete()
+            rec.spec.package.spec._mark_concrete()
 
         self._data = data
 


Test patch: