Repository: spack/spack

Issue:
spack load error: unable to find modulefile
I have successfully installed several spack packages on my Linux Redhat HPC, and I can find the the installations using `spack find`, however the module system does not appear to be working. For example, 

### Steps to reproduce the issue

```console
[sjenness@n2232 ~]$ spack find
==> 36 installed packages
-- linux-centos7-haswell / gcc@4.8.5 ----------------------------
autoconf@2.69              expat@2.2.5       isl@0.19         lmod@8.1.5                 mpfr@3.1.6      readline@7.0
automake@1.16.1            gcc@9.2.0         libbsd@0.9.1     lua@5.3.5                  ncurses@6.1     tar@1.31
bzip2@1.0.8                gdbm@1.18.1       libiconv@1.15    lua-luafilesystem@1_7_0_2  openssl@1.1.1c  tcl@8.6.8
curl@7.63.0                gettext@0.19.8.1  libsigsegv@2.11  lua-luaposix@33.4.0        pcre@8.42       unzip@6.0
diffutils@3.7              git@2.21.0        libtool@2.4.6    m4@1.4.18                  perl@5.26.2     xz@5.2.4
environment-modules@4.3.1  gmp@6.1.2         libxml2@2.9.9    mpc@1.1.0                  pkgconf@1.6.1   zlib@1.2.11
[sjenness@n2232 ~]$ spack find -p gcc
==> 1 installed package
-- linux-centos7-haswell / gcc@4.8.5 ----------------------------
gcc@9.2.0  /gscratch/csde/spack/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/gcc-9.2.0-2ljfzije77bz7bz3b6zwwzhdyulfhvqo
```

### Error Message

But if I try to load any of the packages with `spack load`, I get the following error:
```console
[sjenness@n2232 ~]$ spack load gcc
ERROR: Unable to locate a modulefile for 'gcc-9.2.0-gcc-4.8.5-2ljfzij'
```

Following some old issues here, I have tried `spack module tcl refresh`, but that does not seem to build the module file correctly either. I have not updated any yaml files from their defaults. Thank you!

add generic module directories as well as microarchitecture specific

Currently develop share/spack/setups.{csh,sh} puts just the micro-architecture specific (i.e. ivybridge) paths in `MODULEPATH,` anything built with the generic (i.e. x86_64) architecture thus cannot be accessed via module load, etc.   This patch adds them both to modulepath.  It finds the generic architecture by scanning the output of `spack arch --known-targets`


Discussion:
Can you run `echo $MODULEPATH` and `module avail`?
Please see below. None of my spack module files are listed in module avail, just the ones our HPC admins and contributors have built. Thank you.

```
[sjenness@mox1 ~]$ echo $MODULEPATH
/sw/modules-1.775/modulefiles
[sjenness@mox1 ~]$ module avail
-------------------------------------------------- /sw/modules-1.775/modulefiles --------------------------------------------------
anaconda2_4.3.1                                              contrib/msmc2/2.1.1
anaconda2_5.3                                                contrib/mugsy/1r2.3
anaconda3_4.3.1                                              contrib/multiz/20090121
anaconda3_5.3                                                contrib/multiz/20190527
...
```
In order to use Spack's module files, you need to update your `MODULEPATH` to tell Environment Modules where to find them. If you run `. ~/spack/share/spack/setup-env.sh` or manually set your `MODULEPATH` in your `~/.bashrc`, `module avail` should list all packages built with Spack.
Thank you. I ran `echo $MODULEPATH` above after running `. spack/share/spack/setup-env.sh`, so for some reason it is not getting added automatically anymore (it was as of about 2 weeks ago). 

If I now have to add it manually in `.bashrc` could you please tell the specific syntax I should use (this is not on the spack help pages as far as I can tell)? Thank you, and I'm sorry for what may be a very basic question.
@alalazo can you take a look at this? Spack should be setting up `MODULEPATH` correctly. This may be related to #12916.
I confirm the problem.
I'm not able to reproduce the issue. What I did using 90236bc9f54a0fccbf0e133d09ce258b379e5ae3:
```console
$ module av
module: command not found
$ . share/spack/setup-env.sh 
$ spack arch
linux-ubuntu18.04-broadwell
$ spack bootstrap
[...]
==> Successfully installed environment-modules
  Fetch: 0.01s.  Build: 2.93s.  Total: 2.94s.
[+] /home/culpo/PycharmProjects/spack/opt/spack/linux-ubuntu18.04-broadwell/gcc-9.0.1/environment-modules-4.3.1-fk6yqhgk2sodxzti7cdttfwdy5nbwp2u
$ . share/spack/setup-env.sh
$ module av
------------------------------- /home/culpo/PycharmProjects/spack/share/spack/modules/linux-ubuntu18.04-broadwell --------------------------------
environment-modules-4.3.1-gcc-9.0.1-fk6yqhg  tcl-8.6.8-gcc-9.0.1-utbdhsc  zlib-1.2.11-gcc-9.0.1-os3djb5  
$ spack load zlib
$ module list
Currently Loaded Modulefiles:
 1) zlib-1.2.11-gcc-9.0.1-os3djb5  
```
Can anybody provide more information on version of Spack being used, etc. or better a reliable reproducer to work on?
I might have an idea of what happens. Can people post the output of `spack arch` please?
I think the problem is related to a new naming scheme
spack find now shows me two different sections for gcc@9.1.0:

```
-- linux-centos7-skylake_avx512 / gcc@9.1.0 ---------------------
autoconf@2.69    libiconv@1.15        mpich@3.3.1     readline@7.0
automake@1.16.1  libpciaccess@0.13.5  ncurses@6.1     texinfo@6.5
cmake@2.8.10.2   libsigsegv@2.11      openssl@1.0.2s  util-macros@1.19.1
cmake@3.15.3     libtool@2.4.6        openssl@1.1.1c  xz@5.2.4
findutils@4.6.0  libxml2@2.9.9        perl@5.26.2     zlib@1.2.11
gdbm@1.18.1      m4@1.4.18            pkgconf@1.6.1

-- linux-centos7-x86_64 / gcc@9.1.0 -----------------------------
autoconf@2.69    libsigsegv@2.11  ncurses@6.1    readline@7.0
automake@1.16.1  libtool@2.4.6    perl@5.26.2    texinfo@6.5
gdbm@1.18.1      m4@1.4.18        pkgconf@1.6.1
```

As you can see, we now have the same software for the same system installed twice.

Modulepath only contains the path to the skylake modules, not to the old ones
`/opt/apps/spack/share/spack/modules/linux-centos7-skylake_avx512`
1. Install a package
2. `spack module <> refresh - y` to generate env module 
3. Module not available with `module av` because MODULEPATH is not correctly set/updated by `setup-env.sh`.

I'm not on HEAD right now, maybe it was already fixed meanwhile? Can't check currently 
>     1. Install a package
> 
>     2. `spack module <> refresh - y` to generate env module
> 
>     3. Module not available with `module av` because MODULEPATH is not correctly set/updated by `setup-env.sh`.
> 
> 
> I'm not on HEAD right now, maybe it was already fixed meanwhile? Can't check currently

No, I am at HEAD and it is still not functional
@pat-s That's what is not happening to me :slightly_smiling_face: Can you post the result of both:
```
$ spack find
$ spack arch
```
? My hypothesis is that this happens if you are trying to load modules for architectures that are not the same of your current host.
> Modulepath only contains the path to the skylake modules, not to the old ones
/opt/apps/spack/share/spack/modules/linux-centos7-skylake_avx512

Thanks for the confirmation @oehlrich9 that's exactly what I meant above.
The problem is not only in loading the modules but also in building them

On my system they were previously build into `linux-centos7-x86_64`, now they are build into `linux-centos7-skylake_avx512` on the exact same cluster
This leads to a lot unnecessary compilation as spack does not find already installed packages and builds them again
> @pat-s That's what is not happening to me slightly_smiling_face Can you post the result of both:
> 
> ```
> $ spack find
> $ spack arch
> ```
> 
> ? My hypothesis is that this happens if you are trying to load modules for architectures that are not the same of your current host.

Previously, the architecture was just called x86_64, not it is called skylake_avx512, which makes all previously installed software invalid for that host, even though the architecture is still the same 
@oehlrich9 
> The problem is not only in loading the modules but also in building them

I guess you are using the `develop` branch and update it frequently. In that case the support for specific targets merged a few days ago (#3206) has the effect you mention if you have software already installed at some previous commit. 

I wouldn't consider building optimized software for your architecture as a default a bug though. If you are fine with building generic binaries that can run on any `x86_64` machine you can add this:
```yaml
packages:
  all:
    target: ['x86_64']
```
and that should make `x86_64` the preferred target for the concretizer.

> @oehlrich9
> 
> > The problem is not only in loading the modules but also in building them
> 
> I guess you are using the `develop` branch and update it frequently. In that case the support for specific targets merged a few days ago (#3206) has the effect you mention if you have software already installed at some previous commit.
> 
> I wouldn't consider building optimized software for your architecture as a default a bug though. If you are fine with building generic binaries that can run on any `x86_64` machine you can add this:

Our system is completely homogeneously, so building just x86_64 is fine for us.

> 
> ```yaml
> packages:
>   all:
>     target: ['x86_64']
> ```
> 
> and that should make `x86_64` the preferred target for the concretizer.

Thanks, that did the trick at least for building software
Sadly, `spack load` is still broken and cant find module files
A quick workaround for me is just adding the old modulefile directory manually to the MODULEPATH

```
packages:
  all:
    target: ['x86_64']
```

The setting above solves it.

Since the new microarchitecture approach my arch is `linux-centos7-zen`, before it was `x86_64`.
However a lot of formulas fail for me with this new arch that's why I prefer the old one.
I indeed only see modules for the arch matching `spack arch`, even if I force build packages with another arch, e.g. `x86_64` (just installed one for `linux-centos7-zen`)

So now I suspect `spack module <> refresh` is not generating modules at all for an arch differing from `spack arch`?
A practical problem which is occurring by this is that I am unavailable to load a custom `gcc` because my system compiler always install as `x86_64` and I cannot easily add them to `$MODULEPATH`.

What works is manually call `module use /opt/spack/share/spack/modules/linux-centos7-x86_64`.

So the modules exist, they are just not appended to `$MODULEPATH` if `arch` differs from `spack arch`.
Note: this likely needs to look not only for the generic architecture directory, but also for other compatible microarchitectures.  So we should probably look at everything in `spec.target.ancestors` to see if it exists, and add if so.

Another potential solution would be to just put all the module files in the same directory as before (the generic one), but if you do it that way, you will end up adding too-new packages on older nodes.  I think we probably *do* need to keep these separate as they are now.
> Note: this likely needs to look not only for the generic architecture directory, but also for other compatible microarchitectures. So we should probably look at everything in spec.target.ancestors to see if it exists, and add if so.

This works (or is broken as usual :smirk:) for environment modules. We need some extra care to setup lmod hierarchies if we start mixing microarchitectures. We should probably review also the module files related issues that were introduced by Spack chains and try to come up with a solution that fits both... 

Resolving patch:
diff --git a/lib/spack/spack/architecture.py b/lib/spack/spack/architecture.py
--- a/lib/spack/spack/architecture.py
+++ b/lib/spack/spack/architecture.py
@@ -493,3 +493,15 @@ def sys_type():
     """
     arch = Arch(platform(), 'default_os', 'default_target')
     return str(arch)
+
+
+@memoized
+def compatible_sys_types():
+    """Returns a list of all the systypes compatible with the current host."""
+    compatible_archs = []
+    current_host = cpu.host()
+    compatible_targets = [current_host] + current_host.ancestors
+    for target in compatible_targets:
+        arch = Arch(platform(), 'default_os', target)
+        compatible_archs.append(str(arch))
+    return compatible_archs
diff --git a/lib/spack/spack/main.py b/lib/spack/spack/main.py
--- a/lib/spack/spack/main.py
+++ b/lib/spack/spack/main.py
@@ -585,7 +585,8 @@ def shell_set(var, value):
 
     # print sys type
     shell_set('_sp_sys_type', spack.architecture.sys_type())
-
+    shell_set('_sp_compatible_sys_types',
+              ':'.join(spack.architecture.compatible_sys_types()))
     # print roots for all module systems
     module_roots = spack.config.get('config:module_roots')
     module_to_roots = {


Test patch: