Repository: spack/spack

Issue:
`spack reindex` broke
I see this:

```
$ spack reindex
Traceback (most recent call last):
  File "/Users/eschnett/src/spack/bin/spack", line 179, in <module>
    main()
  File "/Users/eschnett/src/spack/bin/spack", line 157, in main
    return_val = command(parser, args)
  File "/Users/eschnett/src/spack/lib/spack/spack/cmd/reindex.py", line 31, in reindex
    spack.installed_db.reindex(spack.install_layout)
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 316, in reindex
    for spec in directory_layout.all_specs():
  File "/Users/eschnett/src/spack/lib/spack/spack/directory_layout.py", line 301, in all_specs
    return [self.read_spec(s) for s in spec_files]
  File "/Users/eschnett/src/spack/lib/spack/spack/directory_layout.py", line 227, in read_spec
    spec = Spec.from_yaml(f)
  File "/Users/eschnett/src/spack/lib/spack/spack/spec.py", line 1021, in from_yaml
    node[name]['dependencies'].items():
ValueError: too many values to unpack
```



Discussion:
After updating, I now see this:

```
$ spack reindex
Traceback (most recent call last):
  File "/Users/eschnett/src/spack/bin/spack", line 179, in <module>
    main()
  File "/Users/eschnett/src/spack/bin/spack", line 157, in main
    return_val = command(parser, args)
  File "/Users/eschnett/src/spack/lib/spack/spack/cmd/reindex.py", line 31, in reindex
    spack.installed_db.reindex(spack.install_layout)
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 306, in reindex
    with self.write_transaction():
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 612, in __enter__
    self._acquire_fn()
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 382, in _read
    self._read_from_yaml(self._index_path)
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 266, in _read_from_yaml
    self.reindex(spack.install_layout)
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 321, in reindex
    self._check_ref_counts()
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 353, in _check_ref_counts
    (key, found, expected, self._index_path))
AssertionError: Invalid ref_count: m4rjxdobsr22uwld4mhrpwdeviyr57wx: 0 (expected 1), in DB /Users/eschnett/src/spack/opt/spack/.spack-db/index.yaml
```

Some more information:

After reinstalling all packages from scratch, everything was working fine again. I then did `git pull` to update Spack, and this broke `spack reindex` again! I did not know that Spack was so brittle that updating to a new version of some packages would break things.

The errors are:

```
$ spack reindex
Traceback (most recent call last):
  File "/Users/eschnett/src/spack/bin/spack", line 179, in <module>
    main()
  File "/Users/eschnett/src/spack/bin/spack", line 157, in main
    return_val = command(parser, args)
  File "/Users/eschnett/src/spack/lib/spack/spack/cmd/reindex.py", line 31, in reindex
    spack.installed_db.reindex(spack.install_layout)
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 321, in reindex
    self._check_ref_counts()
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 353, in _check_ref_counts
    (key, found, expected, self._index_path))
AssertionError: Invalid ref_count: kcuoucogx2bgk25bvybcd4h3t6o6r5z6: 0 (expected 1), in DB /Users/eschnett/src/spack/opt/spack/.spack-db/index.yaml
```

I also see an error (that I haven't seen before) when re-installing a package:

```
$ env CURL_CA_BUNDLE="$(pwd)/cacert.pem" spack install -v -j8 cactusext +funhpc +julia +simulationio %gcc@6.1.0-spack
==> Installing cactusext
==> hwloc is already installed in /Users/eschnett/src/spack/opt/spack/darwin-elcapitan-x86_64/gcc-6.1.0-spack/hwloc-1.11.3-ungstymua27jvmqsghe4726hmpxsjube
==> git is already installed in /Users/eschnett/src/spack/opt/spack/darwin-elcapitan-x86_64/clang-7.3.0-apple/git-2.8.1-p4rzsh3vic4gwa2ehymvqzos3vppyygv
Traceback (most recent call last):
  File "/Users/eschnett/src/spack/bin/spack", line 179, in <module>
    main()
  File "/Users/eschnett/src/spack/bin/spack", line 157, in main
    return_val = command(parser, args)
  File "/Users/eschnett/src/spack/lib/spack/spack/cmd/install.py", line 91, in install
    explicit=True)
  File "/Users/eschnett/src/spack/lib/spack/spack/package.py", line 910, in do_install
    run_tests=run_tests)
  File "/Users/eschnett/src/spack/lib/spack/spack/package.py", line 1060, in do_install_dependencies
    dep.package.do_install(**kwargs)
  File "/Users/eschnett/src/spack/lib/spack/spack/package.py", line 892, in do_install
    rec = spack.installed_db.get_record(self.spec)
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 76, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 467, in get_record
    key = self._get_matching_spec_key(spec, **kwargs)
  File "/Users/eschnett/src/spack/lib/spack/spack/database.py", line 462, in _get_matching_spec_key
    raise KeyError("No such spec in database! %s" % spec)
KeyError: 'No such spec in database! git@2.8.1%clang@7.3.0-apple arch=darwin-elcapitan-x86_64^autoconf@2.69%clang@7.3.0-apple arch=darwin-elcapitan-x86_64^bzip2@1.0.6%clang@7.3.0-apple arch=darwin-elcapitan-x86_64^cmake@3.6.0%gcc@6.1.0-spack~doc+ncurses+openssl~qt arch=darwin-elcapitan-x86_64^curl@7.50.1%gcc@6.1.0-spack arch=darwin-elcapitan-x86_64^expat@2.1.0%clang@7.3.0-apple arch=darwin-elcapitan-x86_64^gettext@0.19.8.1%clang@7.3.0-apple+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-elcapitan-x86_64^libiconv@1.14%clang@7.3.0-apple arch=darwin-elcapitan-x86_64^libsigsegv@2.10%clang@7.3.0-apple arch=darwin-elcapitan-x86_64^libxml2@2.9.2%clang@7.3.0-apple~python arch=darwin-elcapitan-x86_64^m4@1.4.17%clang@7.3.0-apple+sigsegv arch=darwin-elcapitan-x86_64^ncurses@6.0%gcc@6.1.0-spack arch=darwin-elcapitan-x86_64^openssl@1.0.2h%gcc@6.1.0-spack arch=darwin-elcapitan-x86_64^pcre@8.38%clang@7.3.0-apple+utf arch=darwin-elcapitan-x86_64^perl@5.24.0%clang@7.3.0-apple+cpanm arch=darwin-elcapitan-x86_64^tar@1.29%clang@7.3.0-apple arch=darwin-elcapitan-x86_64^xz@5.2.2%clang@7.3.0-apple arch=darwin-elcapitan-x86_64^zlib@1.2.8%gcc@6.1.0-spack arch=darwin-elcapitan-x86_64'
```

This error is interesting; it seems to point to the fact that a package spec may have existed previously, but now doesn't exist any more after the `git pull`, and Spack gets confused.

This of course is bad. Maybe Spack should interpret already-installed packages via copies of the package descriptions stored with the installed packages? I don't expect Spack to be able to do anything useful with the installed package any more (except uninstall it), but it shouldn't get confused either.


Resolving patch:
diff --git a/lib/spack/spack/database.py b/lib/spack/spack/database.py
--- a/lib/spack/spack/database.py
+++ b/lib/spack/spack/database.py
@@ -436,7 +436,7 @@ def _increment_ref_count(self, spec, directory_layout=None):
 
             self._data[key] = InstallRecord(spec.copy(), path, installed)
 
-            for dep in spec.dependencies('link'):
+            for dep in spec.dependencies(('link', 'run')):
                 self._increment_ref_count(dep)
 
         self._data[key].ref_count += 1


Test patch: