Repository: easybuilders/easybuild-framework

Issue:
long MODULEPATH drastically reduces performance 
I did this test with latest EasyBuld 3.2.1 and Lmod-7.4.5 in a Centos7.3 machine:

With a long `$MODULEPATH` the execution time is quite long:

```
$> echo $MODULEPATH
/scicore/soft/modules/base:/scicore/soft/modules/bio:/scicore/soft/modules/cae:/scicore/soft/modules/chem:/scicore/soft/modules/compiler:/scicore/soft/modules/data:/scicore/soft/modules/debugger:/scicore/soft/modules/devel:/scicore/soft/modules/geo:/scicore/soft/modules/ide:/scicore/soft/modules/lang:/scicore/soft/modules/lib:/scicore/soft/modules/math:/scicore/soft/modules/mpi:/scicore/soft/modules/numlib:/scicore/soft/modules/perf:/scicore/soft/modules/phys:/scicore/soft/modules/system:/scicore/soft/modules/toolchain:/scicore/soft/modules/tools:/scicore/soft/modules/vis

$> time eb foss-2016a.eb -r -D
real	0m57.002s
user	0m36.900s
sys	0m8.828s
```

If I just reduce the `$MODULEPATH` length the execution time is much better:

```
$> export MODULEPATH=/scicore/soft/modules/all

$> time eb foss-2016a.eb -r -D
real	0m8.054s
user	0m4.925s
sys	0m1.361s
```

I don't know if this is the root cause but It seem that when `MODULEPATH` is so long EasyBuild is executing Lmod many more times:

```
$ echo $MODULEPATH 
/scicore/soft/modules/base:/scicore/soft/modules/bio:/scicore/soft/modules/cae:/scicore/soft/modules/chem:/scicore/soft/modules/compiler:/scicore/soft/modules/data:/scicore/soft/modules/debugger:/scicore/soft/modules/devel:/scicore/soft/modules/geo:/scicore/soft/modules/ide:/scicore/soft/modules/lang:/scicore/soft/modules/lib:/scicore/soft/modules/math:/scicore/soft/modules/mpi:/scicore/soft/modules/numlib:/scicore/soft/modules/perf:/scicore/soft/modules/phys:/scicore/soft/modules/system:/scicore/soft/modules/toolchain:/scicore/soft/modules/tools:/scicore/soft/modules/vis

$> time strace -f -o /tmp/eb-strace.log eb foss-2016a.eb -r -D

$> which lmod
/export/soft/lua_lmod/centos7/lmod/lmod/libexec/lmod

$> grep /export/soft/lua_lmod/centos7/lmod/lmod/libexec/lmod /tmp/eb-strace.log | wc -l
948
```



```
$> export MODULEPATH=/scicore/soft/modules/all

$> time strace -f -o /tmp/eb-strace2.log eb foss-2016a.eb -r -D

$> which lmod
/export/soft/lua_lmod/centos7/lmod/lmod/libexec/lmod

$ grep /export/soft/lua_lmod/centos7/lmod/lmod/libexec/lmod /tmp/eb-strace2.log | wc -l
272
```



Discussion:
To make this issue more weird, if I try the same using EB-3.2.1 in Centos6 machine with Lmod-6.0.9 the long `$MODULEPATH` doesn't seem to be a problem:

```
echo $MODULEPATH 
/scicore/soft/modules/base:/scicore/soft/modules/bio:/scicore/soft/modules/chem:/scicore/soft/modules/compiler:/scicore/soft/modules/data:/scicore/soft/modules/devel:/scicore/soft/modules/lang:/scicore/soft/modules/lib:/scicore/soft/modules/math:/scicore/soft/modules/mpi:/scicore/soft/modules/numlib:/scicore/soft/modules/perf:/scicore/soft/modules/phys:/scicore/soft/modules/system:/scicore/soft/modules/toolchain:/scicore/soft/modules/tools:/scicore/soft/modules/vis

$ time eb foss-2016a.eb -r -D
real	0m4.169s
user	0m2.980s
sys	0m0.728s
```




@pescobar That could be because of internal changes to Lmod...

A long `$MODULEPATH` is currently going to make EB run a lot of `module use` commands, which may be handled very differently by Lmod 6 & Lmod 7. Thoughts @rtmclay?
I have a few ideas but I can't really say without more data.  Try timing a module avail and a module load (without the spider cache) for both a long MODULEPATH and a short one.  For the test to be fair, it would be nice if the total number of modules was the same.

Also please do the same thing for Lmod 6.

I think I located the issue. I did this other test in the Centos6-Lmod6.9 environment and I got totally puzzled because I was getting the same bad runtime and the difference in `$MODULEPATH` was 17 folders vs 21 folders:

```
$ echo $MODULEPATH
/scicore/soft/modules/base:/scicore/soft/modules/bio:/scicore/soft/modules/chem:/scicore/soft/modules/compiler:/scicore/soft/modules/data:/scicore/soft/modules/devel:/scicore/soft/modules/lang:/scicore/soft/modules/lib:/scicore/soft/modules/math:/scicore/soft/modules/mpi:/scicore/soft/modules/numlib:/scicore/soft/modules/perf:/scicore/soft/modules/phys:/scicore/soft/modules/system:/scicore/soft/modules/toolchain:/scicore/soft/modules/tools:/scicore/soft/modules/vis

[easybuild@login10 ~]$ time eb foss-2016a.eb -r -D
real	0m4.293s
user	0m2.958s
sys	0m0.809s

export MODULEPATH=/scicore/soft/modules/base:/scicore/soft/modules/bio:/scicore/soft/modules/cae:/scicore/soft/modules/chem:/scicore/soft/modules/compiler:/scicore/soft/modules/data:/scicore/soft/modules/debugger:/scicore/soft/modules/devel:/scicore/soft/modules/geo:/scicore/soft/modules/ide:/scicore/soft/modules/lang:/scicore/soft/modules/lib:/scicore/soft/modules/math:/scicore/soft/modules/mpi:/scicore/soft/modules/numlib:/scicore/soft/modules/perf:/scicore/soft/modules/phys:/scicore/soft/modules/system:/scicore/soft/modules/toolchain:/scicore/soft/modules/tools:/scicore/soft/modules/vis

$> time eb foss-2016a.eb -r -D
real	1m4.156s
user	0m40.182s
sys	0m10.166s
```

Then I realized that my `$MODULEPATH` in Centos7 contains non existing folders like `/scicore/soft/modules/cae` .When I configured Lmod in centos7 I added all the EasyBuild's moduleclasses to the `$MODULEPATH` so in case I add extra modules to those folders in the future the `$MODULEPATH` wouldn't need any update but it seems that this is affecting EasyBuild's or Lmod's performance.

After removing non existing folders from MODULEPATH the performance seems to be much better but I couldn't figure out how the non-existing folders are affecting to the performance

Thanks @boegel and @rtmclay for your help and suggestions!


I'm have 10 directories in `$MODULEPATH` and I count 154 calls to `ml use`. This makes it all very slow.
`module use` supports multiple arguments, that would be an easy way to fix this...
I can also see a big improvement by removing non-existing dirs from MODULEPATH.
```
echo $MODULEPATH
/hpc2n/eb/modules/all/Core:/hpc2n/eb/modules/all/Linux:/hpc2n/eb/software/lmod/lmod/modulefiles/Core
time eb goolfc-2017b.eb --stop=fetch
== temporary log file in case of crash /scratch/eb-Kp_AeF/easybuild-U1V3ni.log
== Core/goolfc/2017b is already installed (module found), skipping
== No easyconfigs left to be built.
== Build succeeded for 0 out of 0
== Temporary log file(s) /scratch/eb-Kp_AeF/easybuild-U1V3ni.log* have been removed.
== Temporary directory /scratch/eb-Kp_AeF has been removed.

real    2m52.153s
user    0m45.004s
sys     0m44.092s

time env MODULEPATH=/hpc2n/eb/modules/all/Core:/hpc2n/eb/software/lmod/lmod/modulefiles/Core eb goolfc-2017b.eb --stop=fetch
== temporary log file in case of crash /scratch/eb-vzFCwJ/easybuild-frRWLR.log
== Core/goolfc/2017b is already installed (module found), skipping
== No easyconfigs left to be built.
== Build succeeded for 0 out of 0
== Temporary log file(s) /scratch/eb-vzFCwJ/easybuild-frRWLR.log* have been removed.
== Temporary directory /scratch/eb-vzFCwJ has been removed.

real    0m7.559s
user    0m3.000s
sys     0m1.804s
```

The /hpc2n/eb/modules/all/Linux dir doesn't exist.

And after creating it...
```
mkdir /hpc2n/eb/modules/all/Linux
b-an02 [goolfc]$ time eb goolfc-2017b.eb --stop=fetch
== temporary log file in case of crash /scratch/eb-9zeVil/easybuild-P2Xcvp.log
== Core/goolfc/2017b is already installed (module found), skipping
== No easyconfigs left to be built.
== Build succeeded for 0 out of 0
== Temporary log file(s) /scratch/eb-9zeVil/easybuild-P2Xcvp.log* have been removed.
== Temporary directory /scratch/eb-9zeVil has been removed.

real    0m7.530s
user    0m3.152s
sys     0m1.720s
```

@rtmclay Does this ring a bell for you? Why would Lmod be significantly slower with a bunch of non-existing paths included in `$MODULEPATH`?

Note that this is with Lmod under EasyBuild's control, so with `$LMOD_IGNORE_CACHE` set...
I encountered the same issue with non-existing directories in `MODULEPATH` slowing down EasyBuild considerably (see #2408, for me the initialization of EasyBuild takes more than 10 minutes). Here is the log: [easybuild-aML_jl.log](https://github.com/easybuilders/easybuild-framework/files/1739092/easybuild-aML_jl.log). Notice how `lmod python use ...` is executed over and over again for the same directories (`grep -o "lmod python use.*ran fine" easybuild-aML_jl.log | sort | uniq -c` says 17 times each).

And maybe the title of this issue should reflect that this only happens with non-existing directories in `MODULEPATH`?
I'm having trouble reproducing this...

I tried adding a whole bunch of non-existing directories in `$MODULEPATH` using:

```
for i in $(seq 1 100); do export MODULEPATH=/tmp/$USER/dir${i}:$MODULEPATH; done
```

But I'm not seeing any noticeable slowdown. Am I overlooking something?
@boegel The issue might be amplified due to our file system having a very bad latency/small read behaviour. Do you have a measurable slow down with non existing directories?
OK, I gave this another try, with:

* `$HOME` an NFS mount (so quite slow)
* 1000 non-existent directories in `$MODULEPATH`, added via `module use`

```
$ ml --version

Modules based on Lua: Version 7.7.26  2018-04-03 23:18 -05:00
    by Robert McLay mclay@tacc.utexas.edu

$ echo $MODULEPATH | wc -c
213
$ time eb foss-2016a.eb -r -D
...
real	0m6.044s
user	0m2.760s
sys	0m1.081s

$ for i in $(seq 1 1000); do echo $i; module use $HOME/nosuchdir${i}; done

$ time eb foss-2016a.eb -r -D
...
real	0m30.824s
user	0m25.835s
sys	0m2.558s
```

So there's a noticeable slowdown of about 5x, but I don't think that's a surprise with 1000 (non-existing) directories, since Lmod will check each of those locations whether it has modules present in those locations.

Creating the directories makes things *a lot* worse, to the point where `eb foss-2016a.eb -r -D` didn't finish in ~1 hour; I guess Lmod tries harder to find modules if the top directory exists.

If someone can still reproduce this, with a recent version of Lmod, please provide debug logs for both a 'fast' run (short `$MODULEPATH`, only existing directories) and a 'slow' run (long `$MODULEPATH` with several non-existing directories).
@boegel I finnaly came around testing this again with the latest versions. I can confirm that it's still the case for me. A dry run takes over 20 minutes if I have non-existing directories in my `MODULEPATH` and if I create all the missing directories it goes down to 20 seconds:

```
# versions involved
$ eb --version
This is EasyBuild 3.6.1 (framework: 3.6.1, easyblocks: 3.6.1) on host node015.
$ ml --version

Modules based on Lua: Version 7.7.34  2018-05-27 17:42 -05:00
    by Robert McLay mclay@tacc.utexas.edu

# clear MODULEPATH
$ unset MODULEPATH
# add all possible module classes
$ ml use $MODULEPATH_EASYBUILD_ROOT/base $MODULEPATH_EASYBUILD_ROOT/bio $MODULEPATH_EASYBUILD_ROOT/cae $MODULEPATH_EASYBUILD_ROOT/chem $MODULEPATH_EASYBUILD_ROOT/compiler $MODULEPATH_EASYBUILD_ROOT/data $MODULEPATH_EASYBUILD_ROOT/debugger $MODULEPATH_EASYBUILD_ROOT/devel $MODULEPATH_EASYBUILD_ROOT/geo $MODULEPATH_EASYBUILD_ROOT/lang $MODULEPATH_EASYBUILD_ROOT/lib $MODULEPATH_EASYBUILD_ROOT/math $MODULEPATH_EASYBUILD_ROOT/mpi $MODULEPATH_EASYBUILD_ROOT/numlib $MODULEPATH_EASYBUILD_ROOT/perf $MODULEPATH_EASYBUILD_ROOT/phys $MODULEPATH_EASYBUILD_ROOT/system $MODULEPATH_EASYBUILD_ROOT/toolchain $MODULEPATH_EASYBUILD_ROOT/tools $MODULEPATH_EASYBUILD_ROOT/vis
# existing directories
$ ls $MODULEPATH_EASYBUILD_ROOT 
all  base  bio  compiler  data  devel  lang  lib  math  mpi  numlib  system  toolchain  tools  vis
# dry-run takes long time
$ time eb --buildpath=/dev/shm --optarch=march=corei7 --prefix=/easybuild/16.04 SAMtools-1.4-foss-2016b.eb -dr -D
== temporary log file in case of crash /tmp/eb-5plycS/easybuild-MdHiPZ.log
Dry run: printing build status of easyconfigs and dependencies
CFGS=/easybuild/16.04/software/EasyBuild/3.6.1/lib/python2.7/site-packages/easybuild_easyconfigs-3.6.1-py2.7.egg/easybuild/easyconfigs
 * [x] $CFGS/m/M4/M4-1.4.17.eb (module: M4/1.4.17)
 * [x] $CFGS/b/Bison/Bison-3.0.4.eb (module: Bison/3.0.4)
 * [x] $CFGS/n/ncurses/ncurses-6.0.eb (module: ncurses/6.0)
 * [x] $CFGS/g/gettext/gettext-0.19.8.eb (module: gettext/0.19.8)
 * [x] $CFGS/f/flex/flex-2.6.0.eb (module: flex/2.6.0)
 * [x] $CFGS/z/zlib/zlib-1.2.8.eb (module: zlib/1.2.8)
 * [x] $CFGS/b/binutils/binutils-2.26.eb (module: binutils/2.26)
 * [x] $CFGS/g/GCCcore/GCCcore-5.4.0.eb (module: GCCcore/5.4.0)
 * [x] $CFGS/m/M4/M4-1.4.17-GCCcore-5.4.0.eb (module: M4/1.4.17-GCCcore-5.4.0)
 * [x] $CFGS/z/zlib/zlib-1.2.8-GCCcore-5.4.0.eb (module: zlib/1.2.8-GCCcore-5.4.0)
 * [x] $CFGS/b/Bison/Bison-3.0.4-GCCcore-5.4.0.eb (module: Bison/3.0.4-GCCcore-5.4.0)
 * [x] $CFGS/f/flex/flex-2.6.0-GCCcore-5.4.0.eb (module: flex/2.6.0-GCCcore-5.4.0)
 * [x] $CFGS/b/binutils/binutils-2.26-GCCcore-5.4.0.eb (module: binutils/2.26-GCCcore-5.4.0)
 * [x] $CFGS/g/GCC/GCC-5.4.0-2.26.eb (module: GCC/5.4.0-2.26)
 * [x] $CFGS/o/OpenBLAS/OpenBLAS-0.2.18-GCC-5.4.0-2.26-LAPACK-3.6.1.eb (module: OpenBLAS/0.2.18-GCC-5.4.0-2.26-LAPACK-3.6.1)
 * [x] $CFGS/m/M4/M4-1.4.17-GCC-5.4.0-2.26.eb (module: M4/1.4.17-GCC-5.4.0-2.26)
 * [x] $CFGS/a/Autoconf/Autoconf-2.69-GCC-5.4.0-2.26.eb (module: Autoconf/2.69-GCC-5.4.0-2.26)
 * [x] $CFGS/a/Automake/Automake-1.15-GCC-5.4.0-2.26.eb (module: Automake/1.15-GCC-5.4.0-2.26)
 * [x] $CFGS/l/libtool/libtool-2.4.6-GCC-5.4.0-2.26.eb (module: libtool/2.4.6-GCC-5.4.0-2.26)
 * [x] $CFGS/a/Autotools/Autotools-20150215-GCC-5.4.0-2.26.eb (module: Autotools/20150215-GCC-5.4.0-2.26)
 * [x] $CFGS/n/numactl/numactl-2.0.11-GCC-5.4.0-2.26.eb (module: numactl/2.0.11-GCC-5.4.0-2.26)
 * [x] $CFGS/h/hwloc/hwloc-1.11.3-GCC-5.4.0-2.26.eb (module: hwloc/1.11.3-GCC-5.4.0-2.26)
 * [x] $CFGS/o/OpenMPI/OpenMPI-1.10.3-GCC-5.4.0-2.26.eb (module: OpenMPI/1.10.3-GCC-5.4.0-2.26)
 * [x] $CFGS/g/gompi/gompi-2016b.eb (module: gompi/2016b)
 * [x] $CFGS/f/FFTW/FFTW-3.3.4-gompi-2016b.eb (module: FFTW/3.3.4-gompi-2016b)
 * [x] $CFGS/s/ScaLAPACK/ScaLAPACK-2.0.2-gompi-2016b-OpenBLAS-0.2.18-LAPACK-3.6.1.eb (module: ScaLAPACK/2.0.2-gompi-2016b-OpenBLAS-0.2.18-LAPACK-3.6.1)
 * [x] $CFGS/f/foss/foss-2016b.eb (module: foss/2016b)
 * [x] $CFGS/b/bzip2/bzip2-1.0.6-foss-2016b.eb (module: bzip2/1.0.6-foss-2016b)
 * [x] $CFGS/m/M4/M4-1.4.17-foss-2016b.eb (module: M4/1.4.17-foss-2016b)
 * [x] $CFGS/l/libtool/libtool-2.4.6-foss-2016b.eb (module: libtool/2.4.6-foss-2016b)
 * [x] $CFGS/n/ncurses/ncurses-6.0-foss-2016b.eb (module: ncurses/6.0-foss-2016b)
 * [x] $CFGS/z/zlib/zlib-1.2.8-foss-2016b.eb (module: zlib/1.2.8-foss-2016b)
 * [x] $CFGS/a/Autoconf/Autoconf-2.69-foss-2016b.eb (module: Autoconf/2.69-foss-2016b)
 * [x] $CFGS/a/Automake/Automake-1.15-foss-2016b.eb (module: Automake/1.15-foss-2016b)
 * [x] $CFGS/a/Autotools/Autotools-20150215-foss-2016b.eb (module: Autotools/20150215-foss-2016b)
 * [x] $CFGS/x/XZ/XZ-5.2.2-foss-2016b.eb (module: XZ/5.2.2-foss-2016b)
 * [ ] $CFGS/s/SAMtools/SAMtools-1.4-foss-2016b.eb (module: SAMtools/1.4-foss-2016b)
== Temporary log file(s) /tmp/eb-5plycS/easybuild-MdHiPZ.log* have been removed.
== Temporary directory /tmp/eb-5plycS has been removed.

real    25m36.102s
user    2m29.416s
sys     3m38.544s
# create all non-existing directories
$ mkdir $MODULEPATH_EASYBUILD_ROOT/cae $MODULEPATH_EASYBUILD_ROOT/chem $MODULEPATH_EASYBUILD_ROOT/debugger $MODULEPATH_EASYBUILD_ROOT/geo $MODULEPATH_EASYBUILD_ROOT/perf $MODULEPATH_EASYBUILD_ROOT/phys
# dry-run is much faster
$ time eb --buildpath=/dev/shm --optarch=march=corei7 --prefix=/easybuild/16.04 SAMtools-1.4-foss-2016b.eb -dr -D
== temporary log file in case of crash /tmp/eb-IJ2KeW/easybuild-6V5fli.log
Dry run: printing build status of easyconfigs and dependencies
CFGS=/easybuild/16.04/software/EasyBuild/3.6.1/lib/python2.7/site-packages/easybuild_easyconfigs-3.6.1-py2.7.egg/easybuild/easyconfigs
 * [x] $CFGS/m/M4/M4-1.4.17.eb (module: M4/1.4.17)
 * [x] $CFGS/b/Bison/Bison-3.0.4.eb (module: Bison/3.0.4)
 * [x] $CFGS/n/ncurses/ncurses-6.0.eb (module: ncurses/6.0)
 * [x] $CFGS/g/gettext/gettext-0.19.8.eb (module: gettext/0.19.8)
 * [x] $CFGS/f/flex/flex-2.6.0.eb (module: flex/2.6.0)
 * [x] $CFGS/z/zlib/zlib-1.2.8.eb (module: zlib/1.2.8)
 * [x] $CFGS/b/binutils/binutils-2.26.eb (module: binutils/2.26)
 * [x] $CFGS/g/GCCcore/GCCcore-5.4.0.eb (module: GCCcore/5.4.0)
 * [x] $CFGS/m/M4/M4-1.4.17-GCCcore-5.4.0.eb (module: M4/1.4.17-GCCcore-5.4.0)
 * [x] $CFGS/z/zlib/zlib-1.2.8-GCCcore-5.4.0.eb (module: zlib/1.2.8-GCCcore-5.4.0)
 * [x] $CFGS/b/Bison/Bison-3.0.4-GCCcore-5.4.0.eb (module: Bison/3.0.4-GCCcore-5.4.0)
 * [x] $CFGS/f/flex/flex-2.6.0-GCCcore-5.4.0.eb (module: flex/2.6.0-GCCcore-5.4.0)
 * [x] $CFGS/b/binutils/binutils-2.26-GCCcore-5.4.0.eb (module: binutils/2.26-GCCcore-5.4.0)
 * [x] $CFGS/g/GCC/GCC-5.4.0-2.26.eb (module: GCC/5.4.0-2.26)
 * [x] $CFGS/o/OpenBLAS/OpenBLAS-0.2.18-GCC-5.4.0-2.26-LAPACK-3.6.1.eb (module: OpenBLAS/0.2.18-GCC-5.4.0-2.26-LAPACK-3.6.1)
 * [x] $CFGS/m/M4/M4-1.4.17-GCC-5.4.0-2.26.eb (module: M4/1.4.17-GCC-5.4.0-2.26)
 * [x] $CFGS/a/Autoconf/Autoconf-2.69-GCC-5.4.0-2.26.eb (module: Autoconf/2.69-GCC-5.4.0-2.26)
 * [x] $CFGS/a/Automake/Automake-1.15-GCC-5.4.0-2.26.eb (module: Automake/1.15-GCC-5.4.0-2.26)
 * [x] $CFGS/l/libtool/libtool-2.4.6-GCC-5.4.0-2.26.eb (module: libtool/2.4.6-GCC-5.4.0-2.26)
 * [x] $CFGS/a/Autotools/Autotools-20150215-GCC-5.4.0-2.26.eb (module: Autotools/20150215-GCC-5.4.0-2.26)
 * [x] $CFGS/n/numactl/numactl-2.0.11-GCC-5.4.0-2.26.eb (module: numactl/2.0.11-GCC-5.4.0-2.26)
 * [x] $CFGS/h/hwloc/hwloc-1.11.3-GCC-5.4.0-2.26.eb (module: hwloc/1.11.3-GCC-5.4.0-2.26)
 * [x] $CFGS/o/OpenMPI/OpenMPI-1.10.3-GCC-5.4.0-2.26.eb (module: OpenMPI/1.10.3-GCC-5.4.0-2.26)
 * [x] $CFGS/g/gompi/gompi-2016b.eb (module: gompi/2016b)
 * [x] $CFGS/f/FFTW/FFTW-3.3.4-gompi-2016b.eb (module: FFTW/3.3.4-gompi-2016b)
 * [x] $CFGS/s/ScaLAPACK/ScaLAPACK-2.0.2-gompi-2016b-OpenBLAS-0.2.18-LAPACK-3.6.1.eb (module: ScaLAPACK/2.0.2-gompi-2016b-OpenBLAS-0.2.18-LAPACK-3.6.1)
 * [x] $CFGS/f/foss/foss-2016b.eb (module: foss/2016b)
 * [x] $CFGS/b/bzip2/bzip2-1.0.6-foss-2016b.eb (module: bzip2/1.0.6-foss-2016b)
 * [x] $CFGS/m/M4/M4-1.4.17-foss-2016b.eb (module: M4/1.4.17-foss-2016b)
 * [x] $CFGS/l/libtool/libtool-2.4.6-foss-2016b.eb (module: libtool/2.4.6-foss-2016b)
 * [x] $CFGS/n/ncurses/ncurses-6.0-foss-2016b.eb (module: ncurses/6.0-foss-2016b)
 * [x] $CFGS/z/zlib/zlib-1.2.8-foss-2016b.eb (module: zlib/1.2.8-foss-2016b)
 * [x] $CFGS/a/Autoconf/Autoconf-2.69-foss-2016b.eb (module: Autoconf/2.69-foss-2016b)
 * [x] $CFGS/a/Automake/Automake-1.15-foss-2016b.eb (module: Automake/1.15-foss-2016b)
 * [x] $CFGS/a/Autotools/Autotools-20150215-foss-2016b.eb (module: Autotools/20150215-foss-2016b)
 * [x] $CFGS/x/XZ/XZ-5.2.2-foss-2016b.eb (module: XZ/5.2.2-foss-2016b)
 * [ ] $CFGS/s/SAMtools/SAMtools-1.4-foss-2016b.eb (module: SAMtools/1.4-foss-2016b)
== Temporary log file(s) /tmp/eb-IJ2KeW/easybuild-6V5fli.log* have been removed.
== Temporary directory /tmp/eb-IJ2KeW has been removed.

real    0m23.276s
user    0m4.188s
sys     0m2.960s
```

How can I tell EasyBuild to keep the logs so I can provide them for further inspection?
@SethosII Please try with `--disable-cleanup-tmpdir`
Here are the [logs](https://github.com/easybuilders/easybuild-framework/files/2089319/logs.zip).

Resolving patch:
diff --git a/easybuild/tools/modules.py b/easybuild/tools/modules.py
--- a/easybuild/tools/modules.py
+++ b/easybuild/tools/modules.py
@@ -342,7 +342,7 @@ def set_mod_paths(self, mod_paths=None):
         # make sure we don't have the same path twice, using nub
         if mod_paths is None:
             # no paths specified, so grab list of (existing) module paths from $MODULEPATH
-            self.mod_paths = [p for p in nub(curr_module_paths()) if os.path.exists(p)]
+            self.mod_paths = nub(curr_module_paths())
         else:
             for mod_path in nub(mod_paths):
                 self.prepend_module_path(mod_path, set_mod_paths=False)
@@ -1475,8 +1475,8 @@ def curr_module_paths():
     """
     Return a list of current module paths.
     """
-    # avoid empty entries, which don't make any sense
-    return [p for p in os.environ.get('MODULEPATH', '').split(':') if p]
+    # avoid empty or nonexistent paths, which don't make any sense
+    return [p for p in os.environ.get('MODULEPATH', '').split(':') if p and os.path.exists(p)]
 
 
 def mk_module_path(paths):


Test patch:
diff --git a/test/framework/modules.py b/test/framework/modules.py
--- a/test/framework/modules.py
+++ b/test/framework/modules.py
@@ -282,6 +282,106 @@ def test_load(self):
         self.assertEqual(os.environ.get('EBROOTGCC'), None)
         self.assertFalse(loaded_modules[-1] == 'GCC/6.4.0-2.28')
 
+    def test_curr_module_paths(self):
+        """Test for curr_module_paths function."""
+
+        # first, create a couple of (empty) directories to use as entries in $MODULEPATH
+        test1 = os.path.join(self.test_prefix, 'test1')
+        mkdir(test1)
+        test2 = os.path.join(self.test_prefix, 'test2')
+        mkdir(test2)
+        test3 = os.path.join(self.test_prefix, 'test3')
+        mkdir(test3)
+
+        os.environ['MODULEPATH'] = ''
+        self.assertEqual(curr_module_paths(), [])
+
+        os.environ['MODULEPATH'] = '%s:%s:%s' % (test1, test2, test3)
+        self.assertEqual(curr_module_paths(), [test1, test2, test3])
+
+        # empty entries and non-existing directories are filtered out
+        os.environ['MODULEPATH'] = '/doesnotexist:%s::%s:' % (test2, test1)
+        self.assertEqual(curr_module_paths(), [test2, test1])
+
+    def test_check_module_path(self):
+        """Test ModulesTool.check_module_path() method"""
+
+        # first, create a couple of (empty) directories to use as entries in $MODULEPATH
+        test1 = os.path.join(self.test_prefix, 'test1')
+        mkdir(test1)
+        test2 = os.path.join(self.test_prefix, 'test2')
+        mkdir(test2)
+        test3 = os.path.join(self.test_prefix, 'test3')
+        mkdir(test3)
+
+        os.environ['MODULEPATH'] = test1
+
+        modtool = modules_tool()
+
+        # directory where modules are installed based on current configuration is automatically added in front
+        mod_install_dir = os.path.join(self.test_installpath, 'modules', 'all')
+        self.assertEqual(modtool.mod_paths, [mod_install_dir, test1])
+
+        # if mod_paths is reset, it can be restored using check_module_path
+        modtool.mod_paths = None
+        modtool.check_module_path()
+        self.assertEqual(modtool.mod_paths, [mod_install_dir, test1])
+
+        # no harm done with multiple subsequent calls
+        modtool.check_module_path()
+        self.assertEqual(modtool.mod_paths, [mod_install_dir, test1])
+
+        # if $MODULEPATH is tweaked, mod_paths and $MODULEPATH can be corrected with check_module_path
+        os.environ['MODULEPATH'] = test2
+        modtool.check_module_path()
+        self.assertEqual(modtool.mod_paths, [mod_install_dir, test1, test2])
+        self.assertEqual(os.environ['MODULEPATH'], mod_install_dir + ':' + test1 + ':' + test2)
+
+        # check behaviour if non-existing directories are included in $MODULEPATH
+        os.environ['MODULEPATH'] = '%s:/does/not/exist:%s' % (test3, test2)
+        modtool.check_module_path()
+        # non-existing dir is filtered from mod_paths, but stays in $MODULEPATH
+        self.assertEqual(modtool.mod_paths, [mod_install_dir, test1, test3, test2])
+        self.assertEqual(os.environ['MODULEPATH'], ':'.join([mod_install_dir, test1, test3, '/does/not/exist', test2]))
+
+    def test_check_module_path_hmns(self):
+        """Test behaviour of check_module_path with HierarchicalMNS."""
+
+        # to verify that https://github.com/easybuilders/easybuild-framework/issues/3084 is fixed
+        # (see also https://github.com/easybuilders/easybuild-framework/issues/2226);
+        # this bug can be triggered by having at least one non-existing directory in $MODULEPATH,
+        # and using HierarchicalMNS
+
+        os.environ['EASYBUILD_MODULE_NAMING_SCHEME'] = 'HierarchicalMNS'
+        init_config()
+
+        top_mod_dir = os.path.join(self.test_installpath, 'modules', 'all')
+        core_mod_dir = os.path.join(top_mod_dir, 'Core')
+        mkdir(core_mod_dir, parents=True)
+
+        doesnotexist = os.path.join(self.test_prefix, 'doesnotexist')
+        self.assertFalse(os.path.exists(doesnotexist))
+
+        os.environ['MODULEPATH'] = '%s:%s' % (core_mod_dir, doesnotexist)
+        modtool = modules_tool()
+
+        self.assertEqual(modtool.mod_paths, [os.path.dirname(core_mod_dir), core_mod_dir])
+        self.assertEqual(os.environ['MODULEPATH'], '%s:%s:%s' % (top_mod_dir, core_mod_dir, doesnotexist))
+
+        # hack prepend_module_path to make sure it's not called again if check_module_path is called again;
+        # prepend_module_path is fairly expensive, so should be avoided,
+        # see https://github.com/easybuilders/easybuild-framework/issues/3084
+        def broken_prepend_module_path(*args, **kwargs):
+            raise EasyBuildError("broken prepend_module_path")
+
+        modtool.prepend_module_path = broken_prepend_module_path
+
+        # if this doesn't trigger a raised error from the hacked prepend_module_path, the bug is fixed
+        modtool.check_module_path()
+
+        self.assertEqual(modtool.mod_paths, [os.path.dirname(core_mod_dir), core_mod_dir])
+        self.assertEqual(os.environ['MODULEPATH'], '%s:%s:%s' % (top_mod_dir, core_mod_dir, doesnotexist))
+
     def test_prepend_module_path(self):
         """Test prepend_module_path method."""
         test_path = tempfile.mkdtemp(prefix=self.test_prefix)