Repository: spack/spack

Issue:
spack commands hanging
After pulling updates there are several spack commands that hang for a really long time. This seems to be related to commit 7ec1d320a82f28e465557046e4b75385bebda77e and #10792.



### Steps to reproduce the issue

```console
$  time spack -d env activate python3_gcc
==> [2019-07-22-21:34:36.817310] Reading config file /opt/packages/gpjohnsn/opt/ssoft/spack/etc/spack/defaults/modules.yaml
==> [2019-07-22-21:34:36.824984] Reading config file /opt/packages/gpjohnsn/opt/ssoft/spack/etc/spack/defaults/linux/modules.yaml
==> [2019-07-22-21:34:36.828422] Reading config file /Users/gpjohnsn/.spack/linux/modules.yaml
==> [2019-07-22-21:34:36.833707] Reading config file /opt/packages/gpjohnsn/opt/ssoft/spack/etc/spack/defaults/config.yaml
==> [2019-07-22-21:34:36.847686] Imported env from built-in commands
==> [2019-07-22-21:34:36.850528] Imported env from built-in commands
==> [2019-07-22-21:36:25.133124] Storing backup of old lockfile /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/environments/python3_gcc/spack.lock at /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/environments/python3_gcc/spack.lock.backup.v1
==> [2019-07-22-21:36:25.134459] Using environmennt 'python3_gcc'

real    1m48.519s
user    1m48.429s
sys     0m0.094s
...
$ time spack -d env status
==> In environment python3_gcc

real    1m51.645s
user    1m51.570s
sys     0m0.080s
```



Discussion:
@glennpj: I can't seem to reproduce this locally.  Can you add your `spack.yaml` and `spack.lock` files to the issue?
@becker33: can you take a look at this one and see if you can replicate it?
I used environment commands as examples but I see it with other commands as well, such as `spack mirror` and `spack stage`.

```console
$ time spack -d stage r-hmisc|& tee spack_hash.txt
==> [2019-07-23-00:38:30.382090] Reading config file /opt/packages/gpjohnsn/opt/ssoft/spack/etc/spack/defaults/modules.yaml
==> [2019-07-23-00:38:30.389500] Reading config file /opt/packages/gpjohnsn/opt/ssoft/spack/etc/spack/defaults/linux/modules.yaml
==> [2019-07-23-00:38:30.392964] Reading config file /Users/gpjohnsn/.spack/linux/modules.yaml
==> [2019-07-23-00:38:30.397948] Reading config file /opt/packages/gpjohnsn/opt/ssoft/spack/etc/spack/defaults/config.yaml
==> [2019-07-23-00:38:30.410051] Imported stage from built-in commands
==> [2019-07-23-00:38:30.410600] Imported stage from built-in commands
==> [2019-07-23-00:38:30.415650] Reading config file /opt/packages/gpjohnsn/opt/ssoft/spack/etc/spack/defaults/repos.yaml
==> [2019-07-23-00:38:31.011520] Reading config file /opt/packages/gpjohnsn/opt/ssoft/spack/etc/spack/defaults/packages.yaml
==> [2019-07-23-00:38:31.027213] READ LOCK: /Users/gpjohnsn/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-07-23-00:38:31.029440] READ LOCK: /Users/gpjohnsn/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-07-23-00:38:31.030031] READ LOCK: /Users/gpjohnsn/.spack/cache/tags/.builtin-index.json.lock[0:0] [Released]
==> [2019-07-23-00:38:31.032787] READ LOCK: /Users/gpjohnsn/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-07-23-00:38:31.033945] READ LOCK: /Users/gpjohnsn/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-07-23-00:38:31.044256] READ LOCK: /Users/gpjohnsn/.spack/cache/patches/.builtin-index.json.lock[0:0] [Released]
==> [2019-07-23-00:38:31.047202] READ LOCK: /Users/gpjohnsn/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-07-23-00:38:31.048204] READ LOCK: /Users/gpjohnsn/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-07-23-00:38:31.069987] READ LOCK: /Users/gpjohnsn/.spack/cache/providers/.builtin-index.json.lock[0:0] [Released]
==> [2019-07-23-00:38:32.410462] Reading config file /Users/gpjohnsn/.spack/linux/compilers.yaml
==> [2019-07-23-00:38:34.599167] Ordered hashes [m4]: m4/16/3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00, m4/19/fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8
==> [2019-07-23-00:38:34.599366] Ordered hashes [perl]: perl/9/0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac
==> [2019-07-23-00:38:34.599666] Ordered hashes [freetype]: freetype/15/08466355e8649235ff01f13b3e56bbd551c7cfb2ca97903cc11575c163ea32a3
==> [2019-07-23-00:38:34.600158] Ordered hashes [glib]: glib/2/c325997b72a205ad1638bb3e3ba0e5b73e3d32ce63b2d0d3282f3e3a2ff4663c
==> [2019-07-23-00:38:34.600330] Ordered hashes [gettext]: gettext/7/9acdb4e73f67c241b5ef32505c9ddf7cf6884ca8ea661692f21dca28483b04b8
==> [2019-07-23-00:38:34.600855] Ordered hashes [gobject-introspection]: gobject-introspection/21/6f90bb267efa043ed70b900b4b8e2faf9e8133afae311893b01060356ea81bba
==> [2019-07-23-00:38:34.601022] Ordered hashes [flex]: flex/22/09c22e5c6fef327d3e48eb23f0d610dcd3a35ab9207f12e0f875701c677978d3
==> [2019-07-23-00:40:49.749840] link /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/stage/r-hmisc-4.1-1-2ia2zbsvsj47fkdx6v37vf25hswwewke -> /tmp/gpjohnsn/spack-stage/spack-stage-y840pb
==> [2019-07-23-00:40:49.751558] Reading config file /Users/gpjohnsn/.spack/linux/mirrors.yaml
==> [2019-07-23-00:40:49.753101] Deprecation warning: md5 checksums will not be supported in future Spack releases.
==> [2019-07-23-00:40:49.755551] Deprecation warning: md5 checksums will not be supported in future Spack releases.
==> [2019-07-23-00:40:49.763253] '/usr/bin/tar' '-xf' '/opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/stage/r-hmisc-4.1-1-2ia2zbsvsj47fkdx6v37vf25hswwewke/Hmisc_4.1-1.tar.gz'
==> [2019-07-23-00:40:49.755139] Using cached archive: /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/cache/r-hmisc/r-hmisc-4.1-1.tar.gz
==> [2019-07-23-00:40:49.762681] Staging archive: /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/stage/r-hmisc-4.1-1-2ia2zbsvsj47fkdx6v37vf25hswwewke/Hmisc_4.1-1.tar.gz
==> [2019-07-23-00:40:49.788653] Created stage in /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/stage/r-hmisc-4.1-1-2ia2zbsvsj47fkdx6v37vf25hswwewke

real    2m19.622s
user    2m19.426s
sys     0m0.212s
```
Most of that time was after
==> [2019-07-23-00:38:34.601022] Ordered hashes [flex]: flex/22/09c22e5c6fef327d3e48eb23f0d610dcd3a35ab9207f12e0f875701c677978d3. I had already downloaded the file so there was no fetching from a remote site.

If I have my git tree at the commit just before the one referenced above then everything works fine.
Can you run one of the slow commands with with `spack -p` and show the profile?
@glennpj: does #10355 fix this for you?  This is based on a hunch but the PR is already done, so if it works we can get it in quickly.
> Can you run one of the slow commands with with spack -p and show the profile?

Yep, I thought of that but had to get some sleep. 😪 

Here is the stage command with the current state:
```console
$ spack -p stage r-hmisc
==> Using cached archive: /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/cache/r-hmisc/r-hmisc-4.1-1.tar.gz
==> Staging archive: /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/stage/r-hmisc-4.1-1-2ia2zbsvsj47fkdx6v37vf25hswwewke/Hmisc_4.1-1.tar.gz
==> Created stage in /opt/packages/gpjohnsn/opt/ssoft/spack/var/spack/stage/r-hmisc-4.1-1-2ia2zbsvsj47fkdx6v37vf25hswwewke
         356136822 function calls (345703070 primitive calls) in 210.438 seconds

   Ordered by: internal time
   List reduced from 1462 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  2165563   19.144    0.000   23.240    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:714(analyze_scalar)
3022273/62888   15.811    0.000  156.637    0.002 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/serializer.py:119(serialize_node)
  2145488   12.834    0.000   17.888    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:1170(write_plain)
65883615/65883614   10.997    0.000   12.791    0.000 {isinstance}
  4130535    9.308    0.000  128.393    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:135(emit)
  1399134    7.177    0.000    9.864    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:668(prepare_tag)
  3022273    6.845    0.000   55.323    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:269(expect_node)
  4233573    6.515    0.000   10.195    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:888(write_indicator)
3022273/62888    5.824    0.000    6.688    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/serializer.py:89(anchor_node)
  8261070    5.571    0.000   14.170    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:146(need_more_events)
  1399134    5.190    0.000   30.577    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:517(check_simple_key)
100823684/100667593    4.975    0.000    5.022    0.000 {len}
  5187989    4.617    0.000    6.137    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/resolver.py:158(resolve)
  8133837    4.224    0.000    4.224    0.000 {method 'encode' of 'unicode' objects}
3022273/62888    4.067    0.000   18.041    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/representer.py:62(represent_data)
  2606089    3.492    0.000    5.806    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:159(need_events)
  2165563    3.212    0.000   21.338    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:597(process_scalar)
  3022273    3.158    0.000   14.951    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:550(process_tag)
  2165563    3.034    0.000   26.963    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:312(expect_scalar)
  1399134    2.928    0.000   45.484    0.000 /opt/packages/gpjohnsn/opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:385(expect_flow_mapping_key)
```
The following is with the commit just before 7ec1d32

```console
$ spack -p stage r-hmisc                                                                    
==> Using cached archive: /opt/ssoft/spack/var/spack/cache/r-hmisc/r-hmisc-4.1-1.tar.gz
==> Staging archive: /opt/ssoft/spack/var/spack/stage/r-hmisc-4.1-1-2ia2zbsvsj47fkdx6v37vf25hswwewke/Hmisc_4.1-1.tar.gz
==> Created stage in /opt/ssoft/spack/var/spack/stage/r-hmisc-4.1-1-2ia2zbsvsj47fkdx6v37vf25hswwewke
         13413231 function calls (12745802 primitive calls) in 7.502 seconds

   Ordered by: internal time
   List reduced from 1462 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
547649/92707    1.017    0.000    1.892    0.000 /opt/ssoft/spack/lib/spack/spack/spec.py:1143(traverse_edges)
   238398    0.274    0.000    1.202    0.000 /opt/ssoft/spack/lib/spack/spack/spec.py:1109(is_virtual)
   170779    0.216    0.000    0.324    0.000 /opt/ssoft/spack/lib/spack/spack/dependency.py:20(canonical_deptype)
   235139    0.207    0.000    0.723    0.000 /opt/ssoft/spack/lib/spack/spack/repo.py:672(exists)
1047505/1047504    0.193    0.000    0.229    0.000 {isinstance}
270695/270294    0.167    0.000    0.369    0.000 {sorted}
   254833    0.161    0.000    0.407    0.000 /opt/ssoft/spack/lib/spack/spack/repo.py:1027(exists)
259842/259838    0.135    0.000    0.234    0.000 /opt/ssoft/spack/lib/spack/llnl/util/lang.py:557(__getattr__)
   270391    0.130    0.000    0.189    0.000 /usr/lib64/python2.7/_abcoll.py:367(__contains__)
    94589    0.116    0.000    0.186    0.000 /usr/lib64/python2.7/_abcoll.py:393(items)
  1211798    0.107    0.000    0.107    0.000 {method 'get' of 'dict' objects}
   470278    0.101    0.000    0.474    0.000 /opt/ssoft/spack/lib/spack/spack/repo.py:677(<genexpr>)
147828/70936    0.099    0.000    0.431    0.000 /opt/ssoft/spack/lib/spack/spack/version.py:85(coercing_method)
    20738    0.097    0.000    0.286    0.000 /opt/ssoft/spack/lib/spack/spack/version.py:98(__init__)
 2375/143    0.095    0.000    3.105    0.022 /opt/ssoft/spack/lib/spack/spack/spec.py:2187(_merge_dependency)
   238398    0.087    0.000    1.289    0.000 /opt/ssoft/spack/lib/spack/spack/spec.py:1098(virtual)
     9655    0.086    0.000    0.105    0.000 /opt/ssoft/spack/lib/spack/external/ruamel/yaml/emitter.py:714(analyze_scalar)
    15842    0.086    0.000    0.096    0.000 /usr/lib64/python2.7/shlex.py:120(read_token)
    89063    0.077    0.000    1.935    0.000 /opt/ssoft/spack/lib/spack/spack/spec.py:1128(traverse)
395995/395994    0.075    0.000    0.562    0.000 {any}
```

I will take a look at #10355 later today, unless you see something that jumps out in the profile runs.
Just a note. The second profile run is from a cloned spack instance. It is in a different directory location from the first but on the same file system.
It looks like we are not caching the hash somewhere we did before. It’s spending all it’s time generating the YAML to be hashed.  So maybe not #10355.  @becker33 is looking into this.

Resolving patch:
diff --git a/lib/spack/spack/hash_types.py b/lib/spack/spack/hash_types.py
--- a/lib/spack/spack/hash_types.py
+++ b/lib/spack/spack/hash_types.py
@@ -18,19 +18,22 @@ class SpecHashDescriptor(object):
 
     We currently use different hashes for different use cases.
     """
-    def __init__(self, deptype=('link', 'run'), package_hash=False):
+    def __init__(self, deptype=('link', 'run'), package_hash=False, attr=None):
         self.deptype = dp.canonical_deptype(deptype)
         self.package_hash = package_hash
+        self.attr = attr
 
 
 #: Default Hash descriptor, used by Spec.dag_hash() and stored in the DB.
-dag_hash = SpecHashDescriptor(deptype=('link', 'run'), package_hash=False)
+dag_hash = SpecHashDescriptor(deptype=('link', 'run'), package_hash=False,
+                              attr='_hash')
 
 
 #: Hash descriptor that includes build dependencies.
 build_hash = SpecHashDescriptor(
-    deptype=('build', 'link', 'run'), package_hash=False)
+    deptype=('build', 'link', 'run'), package_hash=False, attr='_build_hash')
 
 
 #: Full hash used in build pipelines to determine when to rebuild packages.
-full_hash = SpecHashDescriptor(deptype=('link', 'run'), package_hash=True)
+full_hash = SpecHashDescriptor(deptype=('link', 'run'), package_hash=True,
+                               attr='_full_hash')
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1305,7 +1305,7 @@ def _spec_hash(self, hash):
 
         return b32_hash
 
-    def _cached_hash(self, length, attr, hash):
+    def _cached_hash(self, hash, length=None):
         """Helper function for storing a cached hash on the spec.
 
         This will run _spec_hash() with the deptype and package_hash
@@ -1315,13 +1315,16 @@ def _cached_hash(self, length, attr, hash):
         Arguments:
             hash (SpecHashDescriptor): type of hash to generate.
         """
-        hash_string = getattr(self, attr, None)
+        if not hash.attr:
+            return self._spec_hash(hash)[:length]
+
+        hash_string = getattr(self, hash.attr, None)
         if hash_string:
             return hash_string[:length]
         else:
             hash_string = self._spec_hash(hash)
             if self.concrete:
-                setattr(self, attr, hash_string)
+                setattr(self, hash.attr, hash_string)
 
             return hash_string[:length]
 
@@ -1333,7 +1336,7 @@ def dag_hash(self, length=None):
         revise this to include more detailed provenance when the
         concretizer can more aggressievly reuse installed dependencies.
         """
-        return self._cached_hash(length, '_hash', ht.dag_hash)
+        return self._cached_hash(ht.dag_hash, length)
 
     def build_hash(self, length=None):
         """Hash used to store specs in environments.
@@ -1341,7 +1344,7 @@ def build_hash(self, length=None):
         This hash includes build dependencies, and we need to preserve
         them to be able to rebuild an entire environment for a user.
         """
-        return self._cached_hash(length, '_build_hash', ht.build_hash)
+        return self._cached_hash(ht.build_hash, length)
 
     def full_hash(self, length=None):
         """Hash  to determine when to rebuild packages in the build pipeline.
@@ -1352,7 +1355,7 @@ def full_hash(self, length=None):
 
         TODO: investigate whether to include build deps here.
         """
-        return self._cached_hash(length, '_full_hash', ht.full_hash)
+        return self._cached_hash(ht.full_hash, length)
 
     def dag_hash_bit_prefix(self, bits):
         """Get the first <bits> bits of the DAG hash as an integer type."""
@@ -1456,7 +1459,7 @@ def to_node_dict(self, hash=ht.dag_hash):
             d['dependencies'] = syaml_dict([
                 (name,
                  syaml_dict([
-                     ('hash', dspec.spec._spec_hash(hash)),
+                     ('hash', dspec.spec._cached_hash(hash)),
                      ('type', sorted(str(s) for s in dspec.deptypes))])
                  ) for name, dspec in sorted(deps.items())
             ])


Test patch: