Repository: spack/spack

Issue:
module 'spack.hooks' has no attribute 'sbang'
Trying to install some packages from build cache is giving this error:
* `module 'spack.hooks' has no attribute 'sbang'`

Using
* `spack@develop`
* c41782795433cc3241d9a6038c68bcadbb12b838
* `Thu Nov 19 17:43:52 2020 +0100`
* Docker container `ecpe4s/ubuntu18.04-runner-x86_64:2020-09-01`

Concrete spec: [kokkos-p3br7p.yml.txt](https://github.com/spack/spack/files/5569476/kokkos-p3br7p.yml.txt)

Use this `config.yaml`:
```
config:
  install_tree:
    root: /spack-software
    padded_length: 64
```

Setup Spack this way:
```
$> git clone https://github.com/spack/spack /builds/uo/e4s/spack

$> (cd /builds/uo/e4s/spack && git checkout c417827)

$> spack mirror add E4S https://cache.e4s.io

$> wget https://oaciss.uoregon.edu/e4s/e4s.pub

$> spack gpg trust e4s.pub

$> spack config edit config
# use the config.yaml pasted above
```

Install fails:
```
$> spack install --cache-only -f ./kokkos-p3br7p.yml
==> Installing kokkos-3.2.00-2ryuvoz7s4zl7pjmcd5egs6oev2bgcfk
==> Fetching https://cache.e4s.io/build_cache/linux-ubuntu18.04-x86_64/gcc-7.5.0/kokkos-3.2.00/linux-ubuntu18.04-x86_64-gcc-7.5.0-kokkos-3.2.00-2ryuvoz7s4zl7pjmcd5egs6oev2bgcfk.spack
################################################################################################################################################## 100.0%
==> Extracting kokkos-3.2.00-2ryuvoz7s4zl7pjmcd5egs6oev2bgcfk from binary cache
gpgconf: socketdir is '/run/user/0/gnupg'
gpg: Signature made Thu 19 Nov 2020 06:51:21 PM UTC
gpg:                using RSA key 7D344E2992071B0AAAE1EDB0E68DE2A80314303D
gpg: Good signature from "prl" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7D34 4E29 9207 1B0A AAE1  EDB0 E68D E2A8 0314 303D
[+] /spack-software/spack_path_placeholder/spack_path_placeholder/sp/linux-ubuntu18.04-x86_64/gcc-7.5.0/kokkos-3.2.00-2ryuvoz7s4zl7pjmcd5egs6oev2bgcfk
==> Error: Failed to install kokkos due to AttributeError: module 'spack.hooks' has no attribute 'sbang'
==> Error: module 'spack.hooks' has no attribute 'sbang'
```

Install fails w/ stack trace:
```
$> spack -d install --cache-only -f ./kokkos-p3br7p.yml
...
==> [2020-11-19-20:22:34.676428] Flagging kokkos-3.2.00-2ryuvoz7s4zl7pjmcd5egs6oev2bgcfk as failed: module 'spack.hooks' has no attribute 'sbang'
==> [2020-11-19-20:22:34.677820] Error: Failed to install kokkos due to AttributeError: module 'spack.hooks' has no attribute 'sbang'
Traceback (most recent call last):
  File "/builds/uo/e4s/spack/bin/spack", line 66, in <module>
    sys.exit(spack.main.main())
  File "/builds/uo/e4s/spack/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/builds/uo/e4s/spack/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/builds/uo/e4s/spack/lib/spack/spack/cmd/install.py", line 370, in install
    install_specs(args, kwargs, zip(abstract_specs, specs))
  File "/builds/uo/e4s/spack/lib/spack/spack/cmd/install.py", line 212, in install_specs
    builder.install()
  File "/builds/uo/e4s/spack/lib/spack/spack/installer.py", line 1496, in install
    self._install_task(task)
  File "/builds/uo/e4s/spack/lib/spack/spack/installer.py", line 1064, in _install_task
    full_hash_match):
  File "/builds/uo/e4s/spack/lib/spack/spack/installer.py", line 276, in _install_from_cache
    spack.hooks.post_install(pkg.spec)
  File "/builds/uo/e4s/spack/lib/spack/spack/hooks/__init__.py", line 59, in __call__
    hook(*args, **kwargs)
  File "/builds/uo/e4s/spack/lib/spack/spack/hooks/module_file_generation.py", line 30, in post_install
    _for_each_enabled(spec, 'write')
  File "/builds/uo/e4s/spack/lib/spack/spack/hooks/module_file_generation.py", line 22, in _for_each_enabled
    getattr(generator, method_name)()
  File "/builds/uo/e4s/spack/lib/spack/spack/modules/common.py", line 819, in write
    context = self.context.to_dict()
  File "/builds/uo/e4s/spack/lib/spack/spack/tengine.py", line 63, in to_dict
    d = [(name, getattr(self, name)) for name in self.context_properties]
  File "/builds/uo/e4s/spack/lib/spack/spack/tengine.py", line 63, in <listcomp>
    d = [(name, getattr(self, name)) for name in self.context_properties]
  File "/builds/uo/e4s/spack/lib/spack/spack/modules/common.py", line 672, in environment_modifications
    build_environment.set_module_variables_for_package(self.spec.package)
  File "/builds/uo/e4s/spack/lib/spack/spack/build_environment.py", line 539, in set_module_variables_for_package
    _set_variables_for_single_module(pkg, mod)
  File "/builds/uo/e4s/spack/lib/spack/spack/build_environment.py", line 482, in _set_variables_for_single_module
    m.std_cmake_args = spack.build_systems.cmake.CMakePackage._std_args(pkg)
  File "/builds/uo/e4s/spack/lib/spack/spack/build_systems/cmake.py", line 170, in _std_args
    if pkg.spec.satisfies('^cmake@3.9:'):
  File "/builds/uo/e4s/spack/lib/spack/spack/spec.py", line 3108, in satisfies
    return self.satisfies_dependencies(other, strict=deps_strict)
  File "/builds/uo/e4s/spack/lib/spack/spack/spec.py", line 3141, in satisfies_dependencies
    if not self[name].satisfies(other[name], deps=False):
  File "/builds/uo/e4s/spack/lib/spack/spack/spec.py", line 3395, in __getitem__
    return SpecBuildInterface(value, name, query_parameters)
  File "/builds/uo/e4s/spack/lib/spack/spack/spec.py", line 954, in __init__
    is_virtual = spack.repo.path.is_virtual(name)
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 713, in is_virtual
    return have_name and pkg_name in self.provider_index
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 571, in provider_index
    self._provider_index.merge(repo.provider_index)
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 981, in provider_index
    return self.index['providers']
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 403, in __getitem__
    self._build_all_indexes()
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 418, in _build_all_indexes
    self.indexes[name] = self._build_index(name, indexer)
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 448, in _build_index
    indexer.update(namespaced_name)
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 330, in update
    if spack.repo.path.is_virtual(name, use_index=False):
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 716, in is_virtual
    self.get_pkg_class(pkg_name).virtual)
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 679, in get_pkg_class
    return self.repo_for_pkg(pkg_name).get_pkg_class(pkg_name)
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 1126, in get_pkg_class
    module = self._get_pkg_module(pkg_name)
  File "/builds/uo/e4s/spack/lib/spack/spack/repo.py", line 1099, in _get_pkg_module
    prepend=_package_prepend)
  File "/builds/uo/e4s/spack/lib/spack/spack/util/imp/importlib_importer.py", line 48, in load_source
    return loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/builds/uo/e4s/spack/var/spack/repos/builtin/packages/gobject-introspection/package.py", line 8, in <module>

AttributeError: module 'spack.hooks' has no attribute 'sbang'
```

<b>There is no problem if I change SPACK_ROOT to `/builds/uo/e4s/spac` !!! I think this is a big clue... Why would the value of $SPACK_ROOT affect whether or not a package can `import spack.hooks.sbang` ? </b>

Additionally, moving the `import spack.hooks.sbang as sbang` out of global-scope and into method-scope as is done in the following commit *solves this problem* , although I expect this would just push the ball further down the road.
* 7fb300075c00375aa36b6c96a81dc31539094843

@becker33 @scheibelp @tgamblin 


Discussion:
@chuckatkins This is what we were just discussing, fyi.
> There is no problem if I change SPACK_ROOT to /builds/uo/e4s/spac 

A couple questions:

* Where do you change this?
* What is the value of `SPACK_ROOT` before you change it?

Is there any output you are excluding or commands you are running (EDIT: I meant *extra* commands you are running that you didn't mention)? I am currently unable to reproduce the issue although I haven't *exactly* reproduced your conditions (currently using the stock Ubuntu 18.04 image vs. `ecpe4s/ubuntu18.04-runner-x86_64:2020-09-01`) so I'll try to match those more closely.
```
git clone https://github.com/spack/spack ${SPACK_ROOT}
```

Succeeds: `SPACK_ROOT=/builds/uo/e4s/spac` (i.e. Problem is not reproduced)
Fails: `SPACK_ROOT=/builds/uo/e4s/spack` (i.e. Problem *is* reproduced)

The `SPACK_ROOT` is critical to reproducing. I'm not excluding any other commands. The critical piece here is that IF `SPACK_ROOT` is set to `/builds/uo/e4s/spack`, the problem exists. This is the value used to build the binaries in the first place. Does that make sense? Happy to try again with better clarity if not!
Got it: now I can reproduce it
I still haven't gotten to the bottom of this, so I'll be continuing on Monday. 

Todd suggested removing a function-local import of `spack.hooks.sbang` in `relocate.py`,  and I can confirm that resolves the issue:

```
diff --git a/lib/spack/spack/hooks/sbang.py b/lib/spack/spack/hooks/sbang.py
index 687cf651f..4b29a1721 100644
--- a/lib/spack/spack/hooks/sbang.py
+++ b/lib/spack/spack/hooks/sbang.py
@@ -12,7 +12,6 @@
 import llnl.util.tty as tty
 import llnl.util.filesystem as fs
 
-import spack.modules
 import spack.paths
 import spack.store
 
diff --git a/lib/spack/spack/relocate.py b/lib/spack/spack/relocate.py
index 2685b0c6a..64db057cd 100644
--- a/lib/spack/spack/relocate.py
+++ b/lib/spack/spack/relocate.py
@@ -16,6 +16,7 @@
 import spack.repo
 import spack.spec
 import spack.util.executable as executable
+import spack.hooks.sbang
 
 
 class InstallRootStringError(spack.error.SpackError):
@@ -813,9 +814,8 @@ def relocate_text(
     # sbang was a bash script, and it lived in the spack prefix. It is
     # now a POSIX script that lives in the install prefix. Old packages
     # will have the old sbang location in their shebangs.
-    import spack.hooks.sbang as sbang
     orig_sbang = '#!/bin/bash {0}/bin/sbang'.format(orig_spack)
-    new_sbang = sbang.sbang_shebang_line()
+    new_sbang = spack.hooks.sbang.sbang_shebang_line()
 
     # Do relocations on text that refers to the install tree
     for filename in files:
```

As for why this import causes trouble later on, I'm not yet sure.

I've also experimented with a number of different prefixes which all seem to work (without applying the patch):

```
/builds/uo/e4s/spac (as reported above in this issue thread)
/builds/uo/spack
/builds/uo/e4s/subdir/spack
/builds/uo/e4s/flubber
/builds/uo/e4s/spacc
/builds/uo/e4s/spackk
```

I haven't actually been able to create a prefix that *doesn't* work, only `/builds/uo/e4s/spack` (the prefix originally reported in this issue) creates a problem.

I attempted bisecting this which lead to #15702/77b2e578, but without further investigation I can't be sure if that is truly where the problem began: changes to `repo.py` do appear as part of the stack trace, but the function also changes several packages which appear as dependencies of the `kokkos` spec (although the hash of `kokkos` itself is not changed by #15702).

(as a side note for anyone trying to reproduce, make sure to run `spack clean -m`)
I've noticed a couple things that don't quite make sense to me, but may make sense to someone with more Python knowledge.

<b>/spack/lib/spack/spack/hooks/__init__.py</b>
```
from llnl.util.lang import memoized, list_modules
...

@memoized
def all_hook_modules():
  ...
  for name in list_modules(spack.paths.hooks_path)
    ...

...

pre_install = HookRunner('pre_install')
post_install = HookRunner('post_install')
...
```

The following appear to be <b>independent</b> remedies. 

1. Remove `@memoize` decorator from `all_hook_modules()` (defined in <em>hooks/__init__.py</em>)

2. Append `sbang = HookRunner('sbang')` to the end of <em>hooks/__init__.py</em>

3. `import spack.hooks.sbang` such that the import is done in the call stack <b>prior to</b> [Line 34 of <em>hooks/__init__.py</em>](https://github.com/spack/spack/blob/develop/lib/spack/spack/hooks/__init__.py#L34)
* i.e. prior to: `for name in list_modules(spack.paths.hooks_path):`

I believe number 3 explains why we don't see a problem when installing a package from cache into a *different* `SPACK_ROOT` than was used in the build environment. In that case, `relocate.py` includes an `import spack.hooks.sbang` which is invoked prior to [Line 34 of <em>hooks/__init__.py</em>](https://github.com/spack/spack/blob/develop/lib/spack/spack/hooks/__init__.py#L34).

Perhaps someone with a better understanding of Python namespaces or the logic in `llnl.util.lang.memoized` and `llnl.util.lang.list_modules` could make more sense of this.
I get this error too, with v0.16.1 on Cori (Cray), and applying PR #20072 solves it for me
Hit this one too in a seemingly unrelated PR after a rebase, see https://github.com/spack/spack/pull/22720/checks?check_run_id=2359286434
This has cropped up for me again too, this time with `libbsd`.

```
$ spack debug report
* **Spack:** 0.16.1-2274-2a7322c43a
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-haswell
* **Concretizer:** original
```

Steps to reproduce:

```
docker run --rm -ti ubuntu:18.04
apt-get update
apt-get install python3 git build-essential curl
mkdir /work && cd /work
git clone https://github.com/spack/spack.git
cd spack
source share/spack/setup-env.sh
spack -d install libbsd
```

Spack output, including error trace:

```
$ spack -d install libbsd
==> [2021-04-19-19:44:23.306952] Imported install from built-in commands
==> [2021-04-19-19:44:23.310498] Reading config file /work/spack/etc/spack/defaults/config.yaml
==> [2021-04-19-19:44:23.367267] Imported install from built-in commands
==> [2021-04-19-19:44:23.368985] Reading config file /work/spack/etc/spack/defaults/repos.yaml
==> [2021-04-19-19:44:23.545907] Reading config file /work/spack/etc/spack/defaults/packages.yaml
==> [2021-04-19-19:44:23.739290] '/usr/bin/gcc' '--version'
==> [2021-04-19-19:44:23.740193] '/usr/bin/gcc-7' '--version'
==> [2021-04-19-19:44:23.740977] '/usr/bin/g++-7' '--version'
==> [2021-04-19-19:44:23.741816] '/usr/bin/g++' '--version'
==> [2021-04-19-19:44:23.757674] '/usr/bin/g++-7' '-dumpversion'
==> [2021-04-19-19:44:23.759035] '/usr/bin/gcc' '-dumpversion'
==> [2021-04-19-19:44:23.759975] '/usr/bin/g++' '-dumpversion'
==> [2021-04-19-19:44:23.766688] '/usr/bin/gcc-7' '-dumpversion'
==> [2021-04-19-19:44:23.771272] '/usr/bin/gcc' '-dumpfullversion'
==> [2021-04-19-19:44:23.772567] '/usr/bin/g++-7' '-dumpfullversion'
==> [2021-04-19-19:44:23.782222] '/usr/bin/g++' '-dumpfullversion'
==> [2021-04-19-19:44:23.784708] '/usr/bin/gcc-7' '-dumpfullversion'
==> [2021-04-19-19:44:23.795249] Error while executing candidate compiler /usr/bin/cc
TypeError: expected string or bytes-like object
==> [2021-04-19-19:44:23.855437] libbsd applying constraint libmd
==> [2021-04-19-19:44:23.864809] libbsd applying constraint libmd
==> [2021-04-19-19:44:23.877615] DATABASE LOCK TIMEOUT: 3s
==> [2021-04-19-19:44:23.877700] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-04-19-19:44:23.891933] Initializing the build queue from the build requests
==> [2021-04-19-19:44:23.892024] Initializing the build queue for libbsd
==> [2021-04-19-19:44:23.892486] Processing dependencies for libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj: ('build', 'link', 'run')
==> [2021-04-19-19:44:23.892688] Pkg id libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb has the following dependents:
==> [2021-04-19-19:44:23.892762] - libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj
==> [2021-04-19-19:44:23.932742] Pkg id libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj has the following dependents:
==> [2021-04-19-19:44:23.933556] Ensure all dependencies know all dependents across specs
==> [2021-04-19-19:44:23.946035] Acquiring a write lock on libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb with timeout 1e-09
==> [2021-04-19-19:44:23.949407] Creating stage lock spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
==> [2021-04-19-19:44:23.950656] Installing libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
==> [2021-04-19-19:44:23.950751] Searching for binary cache of libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
==> [2021-04-19-19:44:23.950897] Reading config file /work/spack/etc/spack/defaults/mirrors.yaml
==> [2021-04-19-19:44:24.339450] Did not find linux-ubuntu18.04-haswell-gcc-7.5.0-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb.spec.yaml on https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache/linux-ubuntu18.04-haswell-gcc-7.5.0-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb.spec.yaml
  Download failed: HTTP Error 404: Not Found
==> [2021-04-19-19:44:24.339934] No binary for libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb found: installing from source
==> [2021-04-19-19:44:24.353233] '/usr/bin/git' 'describe' '--tags' '--match' 'v*'
==> [2021-04-19-19:44:24.648676] [TOOLCHAIN] MATCH Gcc
==> [2021-04-19-19:44:24.674990] A dependency has updated CPATH, this may lead pkg-config to assume that the package is part of the system includes and omit it when invoked with '--cflags'.
==> [2021-04-19-19:44:24.741623] '/usr/bin/g++' '-v' '/tmp/spack-implicit-link-infopg9cq1db/main.c' '-o' '/tmp/spack-implicit-link-infopg9cq1db/output'
==> [2021-04-19-19:44:24.837328] parsing implicit link info
==> [2021-04-19-19:44:24.838795] linker line:  /usr/lib/gcc/x86_64-linux-gnu/7/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/7/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper -plugin-opt=-fresolution=/tmp/ccQNP5Au.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -z relro -o /tmp/spack-implicit-link-infopg9cq1db/output /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/7/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/7/../../.. /tmp/ccwkhq0E.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crtn.o
==> [2021-04-19-19:44:24.839041] linkdir: /usr/lib/gcc/x86_64-linux-gnu/7
==> [2021-04-19-19:44:24.839143] linkdir: /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu
==> [2021-04-19-19:44:24.839237] linkdir: /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib
==> [2021-04-19-19:44:24.839320] linkdir: /lib/x86_64-linux-gnu
==> [2021-04-19-19:44:24.839403] linkdir: /lib/../lib
==> [2021-04-19-19:44:24.839494] linkdir: /usr/lib/x86_64-linux-gnu
==> [2021-04-19-19:44:24.839595] linkdir: /usr/lib/../lib
==> [2021-04-19-19:44:24.839681] linkdir: /usr/lib/gcc/x86_64-linux-gnu/7/../../..
==> [2021-04-19-19:44:24.839823] found raw link dirs: /usr/lib/gcc/x86_64-linux-gnu/7, /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu, /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib, /lib/x86_64-linux-gnu, /lib/../lib, /usr/lib/x86_64-linux-gnu, /usr/lib/../lib, /usr/lib/gcc/x86_64-linux-gnu/7/../../..
==> [2021-04-19-19:44:24.840094] found link dirs: /usr/lib/gcc/x86_64-linux-gnu/7, /usr/lib/x86_64-linux-gnu, /usr/lib, /lib/x86_64-linux-gnu, /lib
==> [2021-04-19-19:44:24.844590] Checking existence of https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/5a/5a02097f95cc250a3f1001865e4dbba5f1d15554120f95693c0541923c52af4a.tar.xz
==> [2021-04-19-19:44:24.845423] '/usr/bin/curl' '--stderr' '-' '-s' '-f' '-r' '0-0' 'https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/5a/5a02097f95cc250a3f1001865e4dbba5f1d15554120f95693c0541923c52af4a.tar.xz'
==> [2021-04-19-19:44:25.416697] Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/5a/5a02097f95cc250a3f1001865e4dbba5f1d15554120f95693c0541923c52af4a.tar.xz
==> [2021-04-19-19:44:25.418095] '/usr/bin/curl' '-C' '-' '-o' '/tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/libmd-1.0.3.tar.xz.part' '-f' '-D' '-' '-L' 'https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/5a/5a02097f95cc250a3f1001865e4dbba5f1d15554120f95693c0541923c52af4a.tar.xz' '-#' '--connect-timeout' '10'
######################################################################################################################################################################################################### 100.0%
==> [2021-04-19-19:44:26.251208] Staging archive: /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/libmd-1.0.3.tar.xz
==> [2021-04-19-19:44:26.253161] '/bin/tar' '-oxf' '/tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/libmd-1.0.3.tar.xz'
==> [2021-04-19-19:44:26.294094] Created stage in /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
==> [2021-04-19-19:44:26.294869] No patches needed for libmd
==> [2021-04-19-19:44:26.295168] libmd: Building libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb [AutotoolsPackage]
==> [2021-04-19-19:44:26.302460] libmd: Executing phase: 'autoreconf'
==> [2021-04-19-19:44:26.313039] libmd: Executing phase: 'configure'
==> [2021-04-19-19:44:33.879338] libmd: Executing phase: 'build'
==> [2021-04-19-19:44:35.337985] libmd: Executing phase: 'install'
==> [2021-04-19-19:44:35.781371] FileNotFoundError: [Errno 2] No such file or directory: '/work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/repos'
==> [2021-04-19-19:44:35.782316] Installing /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/spack-build-out.txt to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/spack-build-out.txt
==> [2021-04-19-19:44:35.783449] Installing /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/spack-build-01-autoreconf-out.txt to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/spack-build-01-autoreconf-out.txt
==> [2021-04-19-19:44:35.783956] Installing /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/spack-build-02-configure-out.txt to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/spack-build-02-configure-out.txt
==> [2021-04-19-19:44:35.784419] Installing /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/spack-build-03-build-out.txt to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/spack-build-03-build-out.txt
==> [2021-04-19-19:44:35.784866] Installing /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/spack-build-04-install-out.txt to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/spack-build-04-install-out.txt
==> [2021-04-19-19:44:35.785438] Installing /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/spack-build-env.txt to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/spack-build-env.txt
==> [2021-04-19-19:44:35.785937] Installing /tmp/root/spack-stage/spack-stage-libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/spack-configure-args.txt to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/spack-configure-args.txt
==> [2021-04-19-19:44:35.786907] Installing spack-src/config.log to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/archived-files/spack-src/config.log
==> [2021-04-19-19:44:35.787703] Installing spack-src/removed_la_files.txt to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/archived-files/spack-src/removed_la_files.txt
==> [2021-04-19-19:44:35.791279] Installing /work/spack/var/spack/repos/builtin/packages/libmd/package.py to /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb/.spack/repos/builtin/packages/libmd
==> [2021-04-19-19:44:35.792076] Reading config file /work/spack/etc/spack/defaults/modules.yaml
==> [2021-04-19-19:44:35.818374] Reading config file /work/spack/etc/spack/defaults/linux/modules.yaml
==> [2021-04-19-19:44:35.831530] 	WRITE: libmd@1.0.3%gcc@7.5.0 arch=linux-ubuntu18.04-haswell/2aqwdmc [/work/spack/share/spack/modules/linux-ubuntu18.04-haswell/libmd-1.0.3-gcc-7.5.0-2aqwdmc]
==> [2021-04-19-19:44:36.048521] Installing /work/spack/bin/sbang to /work/spack/opt/spack/bin/sbang
==> [2021-04-19-19:44:36.068604] Writing manifest file: No manifest from binary
==> [2021-04-19-19:44:36.089467] libmd: Successfully installed libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
  Fetch: 1.40s.  Build: 9.84s.  Total: 11.25s.
[+] /work/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
==> [2021-04-19-19:44:36.117743] Flagging libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb as installed
==> [2021-04-19-19:44:36.117848] Removing libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb from libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj's uninstalled dependencies.
==> [2021-04-19-19:44:36.118081] libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj: Removed libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb from uninstalled deps list: set()
==> [2021-04-19-19:44:36.118308] Downgrading to a read lock on libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb with timeout 1e-09
==> [2021-04-19-19:44:36.119084] Acquiring a write lock on libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj with timeout 1e-09
==> [2021-04-19-19:44:36.122877] Creating stage lock spack-stage-libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj
==> [2021-04-19-19:44:36.124142] Installing libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj
==> [2021-04-19-19:44:36.124235] Searching for binary cache of libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj
==> [2021-04-19-19:44:36.483202] Did not find linux-ubuntu18.04-haswell-gcc-7.5.0-libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj.spec.yaml on https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache/linux-ubuntu18.04-haswell-gcc-7.5.0-libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj.spec.yaml
  Download failed: HTTP Error 404: Not Found
==> [2021-04-19-19:44:36.483873] No binary for libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj found: installing from source
==> [2021-04-19-19:44:36.511686] '/usr/bin/git' 'describe' '--tags' '--match' 'v*'
==> [2021-04-19-19:44:36.773011] [TOOLCHAIN] MATCH Gcc
==> [2021-04-19-19:44:37.045800] Error: AttributeError: module 'spack.hooks' has no attribute 'sbang'

/work/spack/lib/spack/spack/build_environment.py:930, in _setup_pkg_and_run:
        927        tb_string = traceback.format_exc()
        928
        929        # build up some context from the offending package so we can
  >>    930        # show that, too.
        931        package_context = get_package_context(tb)
        932
        933        logfile = None


Traceback (most recent call last):
  File "/work/spack/lib/spack/spack/build_environment.py", line 913, in _setup_pkg_and_run
    context=context)
  File "/work/spack/lib/spack/spack/build_environment.py", line 794, in setup_package
    set_build_environment_variables(pkg, env, dirty)
  File "/work/spack/lib/spack/spack/build_environment.py", line 379, in set_build_environment_variables
    query = pkg.spec[dep.name]
  File "/work/spack/lib/spack/spack/spec.py", line 3555, in __getitem__
    return SpecBuildInterface(value, name, query_parameters)
  File "/work/spack/lib/spack/spack/spec.py", line 1021, in __init__
    is_virtual = spack.repo.path.is_virtual(name)
  File "/work/spack/lib/spack/spack/repo.py", line 717, in is_virtual
    return have_name and pkg_name in self.provider_index
  File "/work/spack/lib/spack/spack/repo.py", line 575, in provider_index
    self._provider_index.merge(repo.provider_index)
  File "/work/spack/lib/spack/spack/repo.py", line 985, in provider_index
    return self.index['providers']
  File "/work/spack/lib/spack/spack/repo.py", line 407, in __getitem__
    self._build_all_indexes()
  File "/work/spack/lib/spack/spack/repo.py", line 422, in _build_all_indexes
    self.indexes[name] = self._build_index(name, indexer)
  File "/work/spack/lib/spack/spack/repo.py", line 452, in _build_index
    indexer.update(namespaced_name)
  File "/work/spack/lib/spack/spack/repo.py", line 334, in update
    if spack.repo.path.is_virtual(name, use_index=False):
  File "/work/spack/lib/spack/spack/repo.py", line 720, in is_virtual
    self.get_pkg_class(pkg_name).virtual)
  File "/work/spack/lib/spack/spack/repo.py", line 683, in get_pkg_class
    return self.repo_for_pkg(pkg_name).get_pkg_class(pkg_name)
  File "/work/spack/lib/spack/spack/repo.py", line 1130, in get_pkg_class
    module = self._get_pkg_module(pkg_name)
  File "/work/spack/lib/spack/spack/repo.py", line 1103, in _get_pkg_module
    prepend=_package_prepend)
  File "/work/spack/lib/spack/spack/util/imp/importlib_importer.py", line 48, in load_source
    return loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/work/spack/var/spack/repos/builtin/packages/php/package.py", line 9, in <module>
AttributeError: module 'spack.hooks' has no attribute 'sbang'
==> [2021-04-19-19:44:37.046525] Flagging libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj as failed: AttributeError: module 'spack.hooks' has no attribute 'sbang'
==> [2021-04-19-19:44:37.053529] ChildError: AttributeError: module 'spack.hooks' has no attribute 'sbang'
```

I tried the latest work from #20072 and at first it did not cause the problem.  However, I noticed that PR has not been rebased since `libmd` was added as a dependency of `libbsd`, and when I rebased it on the latest `develop`, I can indeed still reproduce the `sbang` issue using the steps listed above.
I'm having trouble reproducing https://github.com/spack/spack/issues/20005#issuecomment-822749016 on a docker Ubuntu 18.04 image (either on develop or on #20072, although FWIW I just merged the `develop` branch into #20072):

```
spack debug report
* **Spack:** 0.16.1-2274-2a7322c43a
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-haswell
* **Concretizer:** original
```

My output:

```
$ spack install libbsd
==> Installing libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
==> No binary for libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb found: installing from source
==> Using cached archive: /root/spack/var/spack/cache/_source-cache/archive/5a/5a02097f95cc250a3f1001865e4dbba5f1d15554120f95693c0541923c52af4a.tar.xz
==> No patches needed for libmd
==> libmd: Executing phase: 'autoreconf'
==> libmd: Executing phase: 'configure'
==> libmd: Executing phase: 'build'
==> libmd: Executing phase: 'install'
==> libmd: Successfully installed libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
  Fetch: 0.00s.  Build: 5.75s.  Total: 5.75s.
[+] /root/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libmd-1.0.3-2aqwdmceft6czeu74d5oospfdi6fpztb
==> Installing libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj
==> No binary for libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj found: installing from source
==> Using cached archive: /root/spack/var/spack/cache/_source-cache/archive/ff/ff95cf8184151dacae4247832f8d4ea8800fa127dbd15033ecfe839f285b42a1.tar.xz
==> Ran patch() for libbsd
==> libbsd: Executing phase: 'autoreconf'
==> libbsd: Executing phase: 'configure'
==> libbsd: Executing phase: 'build'
==> libbsd: Executing phase: 'install'
==> libbsd: Successfully installed libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj
  Fetch: 0.00s.  Build: 11.70s.  Total: 11.71s.
[+] /root/spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/libbsd-0.11.3-7fsouipjoxg4a6nwssxjvitxkb55aacj
```

(and fwiw the spec hashes in this case match yours)

The docker image I'm using:

```
ubuntu                                18.04              ccc6e87d482b
```
Ok, with `056cecf` (the commit where you merged the latest `develop` into your branch for #20072), I can no longer reproduce [#20005 (comment)](https://github.com/spack/spack/issues/20005#issuecomment-822749016).  I'm not sure what happened there, your branch seemed to rebase cleanly on the latest `develop`, but I may just have screwed something up.  Sorry for the noise.
I'm having a similar problem and wanted to check here before I open another/a-new bug.

I've been exploring CI pipelines and have things working to the point where I get buildcache files in the mirror for all of the **Stage-0** jobs.  All of the jobs for **Stage-1** fail, apparently when they're trying to install their dependencies from the **Stage-0** buildcache files.

Debugging this is slow as I need to make a small change to retrigger the pipeline then wade into one of the dynamically generated jobs and click my way through the UI up to the log files stashed within the job's artifacts.

Once I get there, if I check the `pipeline_log.txt` file, I find failures reminiscent of this bug, in this case:

> 'module' object has no attribute 'sbang'

e.g.

```
==> [2021-04-20-23:43:16.876873] Flagging libiconv-1.16-gxhwn4i3pmaieairp27f7dphdysssabc as failed: 'module' object has no attribute 'sbang'
==> [2021-04-20-23:43:16.878062] Warning: Skipping build of diffutils-3.7-eppxpwpiqyhxjyhddhffiucjf364ryl6 since libiconv-1.16-gxhwn4i3pmaieairp27f7dphdysssabc failed
==> [2021-04-20-23:43:16.878124] Flagging diffutils-3.7-eppxpwpiqyhxjyhddhffiucjf364ryl6 as failed
==> [2021-04-20-23:43:16.879336] Error: Failed to install libiconv due to AttributeError: 'module' object has no attribute 'sbang'
```

Quick question, does this seem like this problem or a new unrelated issue?

Deeper question: suggestions on getting things working and/or better debugging harness than rerunning the full CI pipeline?

**Thanks**
I don't know for sure if it's the same problem, but in your CI pipelines, if you have your jobs run `spack ci rebuild`, you could add the `-d` option to get the full stack trace.

As for good workflows for working with CI pipelines, that's a longer discussion to have in a dedicated issue.  I'm interested in that as well, so if you create such an issue, ping me on it :grin: 
@hartzell does #20072 help in your case?
@scottwittenburg -- I am using `-d` and I have lots and lots more output, but none of it seems relevant.

@scheibelp -- I'll give it a try, it'll take a bit because this is an internal image off of an internal mirror of the public repo and (<insert list of moving parts>).
@scheibelp -- #20072 does not appear to help.
@alalazo -- is the commit in your repo worth trying out?  It's unclear if it's on a branch I can grab or ???
@hartzell The commit is from #22720 Not sure why yet, but transforming local imports into imports done at the module level solved the issue for me.
I cherry picked https://github.com/alalazo/spack/commit/a818dcc20f5e7b2b843d2be6afa9b7e1750ebb6b onto the spack develop branch and it seems to have fixed the problem.

Resolving patch:
diff --git a/lib/spack/spack/hooks/__init__.py b/lib/spack/spack/hooks/__init__.py
--- a/lib/spack/spack/hooks/__init__.py
+++ b/lib/spack/spack/hooks/__init__.py
@@ -2,58 +2,72 @@
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
 """This package contains modules with hooks for various stages in the
-   Spack install process.  You can add modules here and they'll be
-   executed by package at various times during the package lifecycle.
-
-   Each hook is just a function that takes a package as a parameter.
-   Hooks are not executed in any particular order.
-
-   Currently the following hooks are supported:
-
-      * pre_install(spec)
-      * post_install(spec)
-      * pre_uninstall(spec)
-      * post_uninstall(spec)
-      * on_install_failure(exception)
-
-   This can be used to implement support for things like module
-   systems (e.g. modules, lmod, etc.) or to add other custom
-   features.
+Spack install process.  You can add modules here and they'll be
+executed by package at various times during the package lifecycle.
+
+Each hook is just a function that takes a package as a parameter.
+Hooks are not executed in any particular order.
+
+Currently the following hooks are supported:
+
+    * pre_install(spec)
+    * post_install(spec)
+    * pre_uninstall(spec)
+    * post_uninstall(spec)
+    * on_install_start(spec)
+    * on_install_success(spec)
+    * on_install_failure(spec)
+    * on_phase_success(pkg, phase_name, log_file)
+    * on_phase_error(pkg, phase_name, log_file)
+    * on_phase_error(pkg, phase_name, log_file)
+    * on_analyzer_save(pkg, result)
+
+This can be used to implement support for things like module
+systems (e.g. modules, lmod, etc.) or to add other custom
+features.
 """
-import os.path
-
+import llnl.util.lang
 import spack.paths
-import spack.util.imp as simp
-from llnl.util.lang import memoized, list_modules
 
 
-@memoized
-def all_hook_modules():
-    modules = []
-    for name in list_modules(spack.paths.hooks_path):
-        mod_name = __name__ + '.' + name
-        path = os.path.join(spack.paths.hooks_path, name) + ".py"
-        mod = simp.load_source(mod_name, path)
-
-        if name == 'write_install_manifest':
-            last_mod = mod
-        else:
-            modules.append(mod)
-
-    # put `write_install_manifest` as the last hook to run
-    modules.append(last_mod)
-    return modules
-
-
-class HookRunner(object):
+class _HookRunner(object):
+    #: Stores all hooks on first call, shared among
+    #: all HookRunner objects
+    _hooks = None
 
     def __init__(self, hook_name):
         self.hook_name = hook_name
 
+    @classmethod
+    def _populate_hooks(cls):
+        # Lazily populate the list of hooks
+        cls._hooks = []
+        relative_names = list(llnl.util.lang.list_modules(
+            spack.paths.hooks_path
+        ))
+
+        # We want this hook to be the last registered
+        relative_names.sort(key=lambda x: x == 'write_install_manifest')
+        assert relative_names[-1] == 'write_install_manifest'
+
+        for name in relative_names:
+            module_name = __name__ + '.' + name
+            # When importing a module from a package, __import__('A.B', ...)
+            # returns package A when 'fromlist' is empty. If fromlist is not
+            # empty it returns the submodule B instead
+            # See: https://stackoverflow.com/a/2725668/771663
+            module_obj = __import__(module_name, fromlist=[None])
+            cls._hooks.append((module_name, module_obj))
+
+    @property
+    def hooks(self):
+        if not self._hooks:
+            self._populate_hooks()
+        return self._hooks
+
     def __call__(self, *args, **kwargs):
-        for module in all_hook_modules():
+        for _, module in self.hooks:
             if hasattr(module, self.hook_name):
                 hook = getattr(module, self.hook_name)
                 if hasattr(hook, '__call__'):
@@ -61,19 +75,19 @@ def __call__(self, *args, **kwargs):
 
 
 # pre/post install and run by the install subprocess
-pre_install = HookRunner('pre_install')
-post_install = HookRunner('post_install')
+pre_install = _HookRunner('pre_install')
+post_install = _HookRunner('post_install')
 
 # These hooks are run within an install subprocess
-pre_uninstall = HookRunner('pre_uninstall')
-post_uninstall = HookRunner('post_uninstall')
-on_phase_success = HookRunner('on_phase_success')
-on_phase_error = HookRunner('on_phase_error')
+pre_uninstall = _HookRunner('pre_uninstall')
+post_uninstall = _HookRunner('post_uninstall')
+on_phase_success = _HookRunner('on_phase_success')
+on_phase_error = _HookRunner('on_phase_error')
 
 # These are hooks in installer.py, before starting install subprocess
-on_install_start = HookRunner('on_install_start')
-on_install_success = HookRunner('on_install_success')
-on_install_failure = HookRunner('on_install_failure')
+on_install_start = _HookRunner('on_install_start')
+on_install_success = _HookRunner('on_install_success')
+on_install_failure = _HookRunner('on_install_failure')
 
 # Analyzer hooks
-on_analyzer_save = HookRunner('on_analyzer_save')
+on_analyzer_save = _HookRunner('on_analyzer_save')


Test patch: