Repository: spack/spack

Issue:
NoSuchPatchError for graphviz, unable to build anything
It seems like my Spack database got corrupted somehow? Anytime I try to build any package, I see:
```
==> Error: Failed to install re2c due to NoSuchPatchError: Couldn't find patch for package builtin.graphviz with sha256: 99962eb1a491bbaa8afc6d8d238f3edb3b45badf1c1f9af026ed52c6c28d2d8a
```
even for packages that don't depend on `graphviz`. I tried both `spack clean -m` and `spack clean -p` and neither fixed it. `spack reindex` crashes with the same error.

### Steps to reproduce the issue

```console
$ spack reindex
$ spack install re2c
```

### Error Message

<!-- If Spack reported an error, provide the error message. If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->
```console
$ spack -d install re2c
...
  Fetch: 0.02s.  Build: 1m 40.30s.  Total: 1m 40.32s.
[+] /Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/re2c-1.2.1-7eg6bjrpxpiackuur7ruknhsbjz2wikq
==> [2020-10-31-21:56:25.779818] NoSuchPatchError: Couldn't find patch for package builtin.graphviz with sha256: 99962eb1a491bbaa8afc6d8d238f3edb3b45badf1c1f9af026ed52c6c28d2d8a
==> [2020-10-31-21:56:25.780602] Error: Failed to install re2c due to NoSuchPatchError: Couldn't find patch for package builtin.graphviz with sha256: 99962eb1a491bbaa8afc6d8d238f3edb3b45badf1c1f9af026ed52c6c28d2d8a
==> [2020-10-31-21:56:25.780758] Flagging re2c as failed: Couldn't find patch for package builtin.graphviz with sha256: 99962eb1a491bbaa8afc6d8d238f3edb3b45badf1c1f9af026ed52c6c28d2d8a
==> [2020-10-31-21:56:25.788636] NoSuchPatchError: Couldn't find patch for package builtin.graphviz with sha256: 99962eb1a491bbaa8afc6d8d238f3edb3b45badf1c1f9af026ed52c6c28d2d8a
==> [2020-10-31-21:56:25.788763] Error: Couldn't find patch for package builtin.graphviz with sha256: 99962eb1a491bbaa8afc6d8d238f3edb3b45badf1c1f9af026ed52c6c28d2d8a
Traceback (most recent call last):
  File "/Users/Adam/spack/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/Users/Adam/spack/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/Users/Adam/spack/lib/spack/spack/cmd/install.py", line 405, in install
    install_spec(args, kwargs, abstract, concrete)
  File "/Users/Adam/spack/lib/spack/spack/cmd/install.py", line 256, in install_spec
    spec.package.do_install(**kwargs)
  File "/Users/Adam/spack/lib/spack/spack/package.py", line 1602, in do_install
    builder.install(**kwargs)
  File "/Users/Adam/spack/lib/spack/spack/installer.py", line 1591, in install
    self._install_task(task, **kwargs)
  File "/Users/Adam/spack/lib/spack/spack/installer.py", line 1210, in _install_task
    explicit=explicit)
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 120, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 1135, in add
    self._add(spec, directory_layout, explicit=explicit)
  File "/Users/Adam/spack/lib/spack/llnl/util/lock.py", line 584, in __exit__
    if self._exit(release_fn):
  File "/Users/Adam/spack/lib/spack/llnl/util/lock.py", line 605, in _exit
    return self._lock.release_write(release_fn)
  File "/Users/Adam/spack/lib/spack/llnl/util/lock.py", line 461, in release_write
    result = release_fn()
  File "/Users/Adam/spack/lib/spack/llnl/util/lock.py", line 578, in release_fn
    return self._release_fn(type, value, traceback)
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 978, in _write
    self._write_to_file(f)
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 613, in _write_to_file
    for k, v in self._data.items())
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 613, in <genexpr>
    for k, v in self._data.items())
  File "/Users/Adam/spack/lib/spack/spack/database.py", line 212, in to_dict
    rec_dict.update({'spec': self.spec.node_dict_with_hashes()})
  File "/Users/Adam/spack/lib/spack/spack/spec.py", line 1654, in node_dict_with_hashes
    node[self.name]['full_hash'] = self.full_hash()
  File "/Users/Adam/spack/lib/spack/spack/spec.py", line 1464, in full_hash
    return self._cached_hash(ht.full_hash, length)
  File "/Users/Adam/spack/lib/spack/spack/spec.py", line 1431, in _cached_hash
    hash_string = self._spec_hash(hash)
  File "/Users/Adam/spack/lib/spack/spack/spec.py", line 1404, in _spec_hash
    node_dict = self.to_node_dict(hash=hash)
  File "/Users/Adam/spack/lib/spack/spack/spec.py", line 1562, in to_node_dict
    d['package_hash'] = self.package.content_hash()
  File "/Users/Adam/spack/lib/spack/spack/package.py", line 1437, in content_hash
    for p in self.spec.patches)
  File "/Users/Adam/spack/lib/spack/llnl/util/lang.py", line 197, in _memoized_function
    func.cache[args] = func(*args)
  File "/Users/Adam/spack/lib/spack/spack/spec.py", line 3090, in patches
    patch = index.patch_for_package(sha256, self.package)
  File "/Users/Adam/spack/lib/spack/spack/patch.py", line 350, in patch_for_package
    % (pkg.fullname, sha256))
spack.patch.NoSuchPatchError: Couldn't find patch for package builtin.graphviz with sha256: 99962eb1a491bbaa8afc6d8d238f3edb3b45badf1c1f9af026ed52c6c28d2d8a
```

### Information on your system

<!-- Please include the output of `spack debug report` -->
* **Spack:** 0.15.4-1753-51a9473bf8
* **Python:** 3.8.5
* **Platform:** darwin-catalina-ivybridge

I also tried with Python 2 but same error.
<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
For anyone else who encounters this, the current workaround is to `rm -rf` the installation directory of the offending package (in my case, graphviz) and run `spack reindex`.

@aweits reported that this is likely due to https://github.com/spack/spack/pull/19209. @scottwittenburg?

Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1011,6 +1011,14 @@ def __init__(self, spec_like=None,
         self.external_modules = Spec._format_module_list(external_modules)
         self._full_hash = full_hash
 
+        # Older spack versions did not compute full_hash or build_hash,
+        # and we may not have the necessary information to recompute them
+        # if we read in old specs. Old concrete specs are marked "final"
+        # when read in to indicate that we shouldn't recompute full_hash
+        # or build_hash. New specs are not final; we can lazily compute
+        # their hashes.
+        self._hashes_final = False
+
         # This attribute is used to store custom information for
         # external specs. None signal that it was not set yet.
         self.extra_attributes = None
@@ -1650,10 +1658,33 @@ def node_dict_with_hashes(self, hash=ht.dag_hash):
         the hash_type, the build hash is also added. """
         node = self.to_node_dict(hash)
         node[self.name]['hash'] = self.dag_hash()
+
+        # full_hash and build_hash are lazily computed -- but if we write
+        # a spec out, we want them to be included. This is effectively
+        # the last chance we get to compute them accurately.
         if self.concrete:
-            node[self.name]['full_hash'] = self.full_hash()
-        if 'build' in hash.deptype:
-            node[self.name]['build_hash'] = self.build_hash()
+            # build and full hashes can be written out if:
+            # 1. they're precomputed (i.e. we read them from somewhere
+            #    and they were already on the spec
+            # 2. we can still compute them lazily (i.e. we just made them and
+            #    have the full dependency graph on-hand)
+            #
+            # we want to avoid recomputing either hash for specs we read
+            # in from the DB or elsewhere, as we may not have the info
+            # (like patches, package versions, etc.) that we need to
+            # compute them. Unknown hashes are better than wrong hashes.
+            write_full_hash = (
+                self._hashes_final and self._full_hash or   # cached and final
+                not self._hashes_final)                     # lazily compute
+            write_build_hash = 'build' in hash.deptype and (
+                self._hashes_final and self._build_hash or  # cached and final
+                not self._hashes_final)                     # lazily compute
+
+            if write_full_hash:
+                node[self.name]['full_hash'] = self.full_hash()
+            if write_build_hash:
+                node[self.name]['build_hash'] = self.build_hash()
+
         return node
 
     def to_record_dict(self):
@@ -1743,6 +1774,11 @@ def from_node_dict(node):
         # specs read in are concrete unless marked abstract
         spec._concrete = node.get('concrete', True)
 
+        # this spec may have been built with older packages than we have
+        # on-hand, and we may not have the build dependencies, so mark it
+        # so we don't recompute full_hash and build_hash.
+        spec._hashes_final = spec._concrete
+
         if 'patches' in node:
             patches = node['patches']
             if len(patches) > 0:
@@ -3176,6 +3212,7 @@ def _dup(self, other, deps=True, cleardeps=True, caches=None):
             self._dup_deps(other, deptypes, caches)
 
         self._concrete = other._concrete
+        self._hashes_final = other._hashes_final
 
         if caches:
             self._hash = other._hash


Test patch: