Repository: spack/spack

Issue:
spack load is slow on GPFS
I'm often on JUSUF at JSC which is backed by IBM's GPFS. However I notice `spack load` is slow.

```
$ time spack load llvm /zbn

real	0m13.219s
user	0m1.130s
sys	0m1.060s
```

I am not in an environment. 


Discussion:
Could @alalazo want to take a look at this? Do you have access to the parallel FS systems?
I wonder if this is what others are also seeing.
@ikitayama Thanks for the report. I don't have access to JUSUF, but can access other systems with GPFS.
@alalazo do you happen to know the Spack profile option, any chance?
@ikitayama I'll very likely get to this after the Spack BoF next week i.e. after we released Spack v0.16
Good. In the mean time (I won’t be around the bof next week), I’ll keep
updating develop branch and see if anything comes up.

On Sat, Nov 14, 2020 at 17:10 Massimiliano Culpo <notifications@github.com>
wrote:

> @ikitayama <https://github.com/ikitayama> I'll very likely get to this
> after the Spack BoF next week i.e. after we released Spack v0.16
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/spack/spack/issues/19871#issuecomment-727164581>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHLBZ7EUQ3DPDPRLIG5LMDSPY3QDANCNFSM4TSYO3FQ>
> .
>

I'll be away from the office the entire next week, but check my inbox from time to time, so if you'd want me to try some commands let me know.
Welcome to 2021, as of this writing, it takes a few minutes to log on to JURECA-DC as I have my .bash_profile containing a line:

`spack load llvm /biqe`

... Hmm.
@alalazo your expert help is much appreciated.
I am reporting that a single Spack package (llvm) load described in my bash login script prevents me from logging on
to JURECA-DC login node; not only it takes time.
May I also ask @tgamblin and @becker33 to take a look at this real issue? If the work-around is just 'don't load anything
at login time, I'd follow that advice.
On a non-GPFS system, Spack operates much more quickly, when loading a spec. I'd like to see  an improvement for the GPFS or parallel filesystems in general. @alalazo can you tackle on this issue again if you have time. 
Please note I've been on the cutting edge branch; develop for a long time. 
Can you enable profiling and see where most time is spent @ikitayama?

```
spack -p load --sh llvm
```
Will do that when JURECA comes back online (tomorrow your time)!

On Wed, Apr 14, 2021 at 5:16 PM Harmen Stoppels
***@***.***> wrote:
>
> Can you enable profiling and see where most time is spent @ikitayama?
>
> spack -p load --sh llvm
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Oops the day after tomorrow

On Wed, Apr 14, 2021 at 17:18 Itaru Kitayama ***@***.***>
wrote:

> Will do that when JURECA comes back online (tomorrow your time)!
>
> On Wed, Apr 14, 2021 at 5:16 PM Harmen Stoppels
> ***@***.***> wrote:
> >
> > Can you enable profiling and see where most time is spent @ikitayama?
> >
> > spack -p load --sh llvm
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub, or unsubscribe.
>

> Can you enable profiling and see where most time is spent @ikitayama?
> 
> ```
> spack -p load --sh llvm
> ```

The data are taken on JURECA-DC:
```
$ spack -p load --sh llvm /p3m
export MANPATH=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/python-3.8.8-imhoumrie4wglmjsmlu3ovzmh2hkiayx/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/util-linux-uuid-2.36-ongln3u6sqqymu7aspiqsi76qvvp4koq/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/sqlite-3.34.0-7io6mp5ax2ddpaycp364quaonrpuffaj/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gdbm-1.19-tsjaztspbf2z4aiyu56nwiwnfklo6nlx/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/readline-8.0-5knyycilfqlyjjntb7yjdxmu327w7dkl/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libbsd-0.10.0-ex3xe7rxzmjcnikxkqbzaxybw7vmzi2q/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/pcre-8.44-ihgobdpfkpx3jbpxxw7xwx2gip4jjls6/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libffi-3.3-tzxzh4ht5w3cijtjmn4ughfr7llzrojx/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libedit-3.1-20210216-fcegcr5md4lua5ahuqhrh462cw7pmaul/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/hwloc-2.4.1-quyjzlvk6gc5dtq5i4keo5nchaceymrf/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/binutils-2.36.1-shxl54zbnjrlidbigrf3hsu3ywduplbl/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gettext-0.21-2zusk3zerjuwltyzns7rlpqgdih6y4w5/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/tar-1.32-x3qdbp3hbpb7lvhmwrpdjcuyo2ommzaw/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libxml2-2.9.10-lddvpplxexarse3lxzmalchfdk7gtz4r/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/zlib-1.2.11-uw42aiy6225mpwyeha5txnsde4pxaey2/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/xz-5.2.5-tnkfg2wegm2rnfqdwfnroclzam4qrxca/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libiconv-1.16-zmywoh3dwchbn67szay6bztarri4g3t2/share/man:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/bzip2-1.0.8-72rw76qwcjufmomjqdldjilz2oeecjrx/man:/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/share/man:/p/software/jurecadc/stages/2020/software/zlib/1.2.11-GCCcore-9.3.0/share/man:/p/software/jurecadc/stages/2020/software/GCCcore/9.3.0/share/man:/p/software/jurecadc/lmod/lmod/share/man:/usr/local/share/man:/usr/share/man:/opt/ddn/ime/share/man:/opt/ddn/ime/share/man;
export CMAKE_PREFIX_PATH=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/z3-4.8.9-nnnpaka5o43bqefkytx4begoe4xjtiil:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/py-setuptools-50.3.2-wk4ghswz5tx7az26ky2pxr6ieooslzg7:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/python-3.8.8-imhoumrie4wglmjsmlu3ovzmh2hkiayx:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/util-linux-uuid-2.36-ongln3u6sqqymu7aspiqsi76qvvp4koq:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/sqlite-3.34.0-7io6mp5ax2ddpaycp364quaonrpuffaj:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/openssl-1.1.1k-s6szyx5khijyg6drni2yaharjcnmkau3:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gdbm-1.19-tsjaztspbf2z4aiyu56nwiwnfklo6nlx:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/readline-8.0-5knyycilfqlyjjntb7yjdxmu327w7dkl:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/expat-2.2.10-grguipxxnciycsmllwr6myjmpajxbkqz:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libbsd-0.10.0-ex3xe7rxzmjcnikxkqbzaxybw7vmzi2q:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/swig-4.0.2-zsjwbhzfaqh3swtvasl7kdadrsp5my4m:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/pcre-8.44-ihgobdpfkpx3jbpxxw7xwx2gip4jjls6:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libffi-3.3-tzxzh4ht5w3cijtjmn4ughfr7llzrojx:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libelf-0.8.13-styifuipry54smq2ambvymmtasdpju75:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libedit-3.1-20210216-fcegcr5md4lua5ahuqhrh462cw7pmaul:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/hwloc-2.4.1-quyjzlvk6gc5dtq5i4keo5nchaceymrf:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libpciaccess-0.16-jml4hzwgnmtps5wh3wf5cy7bxxl6d673:/p/software/jurecadc/stages/2020/software/CUDA/11.0:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/binutils-2.36.1-shxl54zbnjrlidbigrf3hsu3ywduplbl:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gettext-0.21-2zusk3zerjuwltyzns7rlpqgdih6y4w5:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/tar-1.32-x3qdbp3hbpb7lvhmwrpdjcuyo2ommzaw:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/ncurses-6.2-hznv2jsuev33sr4kgpyjdl6tbycbhmwt:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libxml2-2.9.10-lddvpplxexarse3lxzmalchfdk7gtz4r:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/zlib-1.2.11-uw42aiy6225mpwyeha5txnsde4pxaey2:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/xz-5.2.5-tnkfg2wegm2rnfqdwfnroclzam4qrxca:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libiconv-1.16-zmywoh3dwchbn67szay6bztarri4g3t2:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/bzip2-1.0.8-72rw76qwcjufmomjqdldjilz2oeecjrx:/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0:/p/software/jurecadc/stages/2020/software/zlib/1.2.11-GCCcore-9.3.0:/p/software/jurecadc/stages/2020/software/GCCcore/9.3.0;
export PYTHONPATH=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25/lib/python3.8/site-packages:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/z3-4.8.9-nnnpaka5o43bqefkytx4begoe4xjtiil/lib/python3.8/site-packages:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/py-setuptools-50.3.2-wk4ghswz5tx7az26ky2pxr6ieooslzg7/lib/python3.8/site-packages:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/py-setuptools-50.3.2-wk4ghswz5tx7az26ky2pxr6ieooslzg7/lib/python3.8/site-packages:/p/project/cjzam11/kitayama1/opt/nrn-master/lib/python:;
export LD_LIBRARY_PATH=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/z3-4.8.9-nnnpaka5o43bqefkytx4begoe4xjtiil/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/py-setuptools-50.3.2-wk4ghswz5tx7az26ky2pxr6ieooslzg7/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/python-3.8.8-imhoumrie4wglmjsmlu3ovzmh2hkiayx/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/util-linux-uuid-2.36-ongln3u6sqqymu7aspiqsi76qvvp4koq/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/sqlite-3.34.0-7io6mp5ax2ddpaycp364quaonrpuffaj/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/openssl-1.1.1k-s6szyx5khijyg6drni2yaharjcnmkau3/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gdbm-1.19-tsjaztspbf2z4aiyu56nwiwnfklo6nlx/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/readline-8.0-5knyycilfqlyjjntb7yjdxmu327w7dkl/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/expat-2.2.10-grguipxxnciycsmllwr6myjmpajxbkqz/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libbsd-0.10.0-ex3xe7rxzmjcnikxkqbzaxybw7vmzi2q/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/pcre-8.44-ihgobdpfkpx3jbpxxw7xwx2gip4jjls6/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libffi-3.3-tzxzh4ht5w3cijtjmn4ughfr7llzrojx/lib64:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libffi-3.3-tzxzh4ht5w3cijtjmn4ughfr7llzrojx/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libelf-0.8.13-styifuipry54smq2ambvymmtasdpju75/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libedit-3.1-20210216-fcegcr5md4lua5ahuqhrh462cw7pmaul/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/hwloc-2.4.1-quyjzlvk6gc5dtq5i4keo5nchaceymrf/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libpciaccess-0.16-jml4hzwgnmtps5wh3wf5cy7bxxl6d673/lib:/p/software/jurecadc/stages/2020/software/CUDA/11.0/lib64:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/binutils-2.36.1-shxl54zbnjrlidbigrf3hsu3ywduplbl/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gettext-0.21-2zusk3zerjuwltyzns7rlpqgdih6y4w5/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/ncurses-6.2-hznv2jsuev33sr4kgpyjdl6tbycbhmwt/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libxml2-2.9.10-lddvpplxexarse3lxzmalchfdk7gtz4r/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/zlib-1.2.11-uw42aiy6225mpwyeha5txnsde4pxaey2/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/xz-5.2.5-tnkfg2wegm2rnfqdwfnroclzam4qrxca/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libiconv-1.16-zmywoh3dwchbn67szay6bztarri4g3t2/lib:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/bzip2-1.0.8-72rw76qwcjufmomjqdldjilz2oeecjrx/lib:/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/lib:/p/software/jurecadc/stages/2020/software/zlib/1.2.11-GCCcore-9.3.0/lib:/p/software/jurecadc/stages/2020/software/GCCcore/9.3.0/lib64:/p/software/jurecadc/stages/2020/software/GCCcore/9.3.0/lib;
export F77=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25/bin/flang;
export PKG_CONFIG_PATH=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/python-3.8.8-imhoumrie4wglmjsmlu3ovzmh2hkiayx/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/util-linux-uuid-2.36-ongln3u6sqqymu7aspiqsi76qvvp4koq/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/sqlite-3.34.0-7io6mp5ax2ddpaycp364quaonrpuffaj/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/openssl-1.1.1k-s6szyx5khijyg6drni2yaharjcnmkau3/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/readline-8.0-5knyycilfqlyjjntb7yjdxmu327w7dkl/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/expat-2.2.10-grguipxxnciycsmllwr6myjmpajxbkqz/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libbsd-0.10.0-ex3xe7rxzmjcnikxkqbzaxybw7vmzi2q/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/pcre-8.44-ihgobdpfkpx3jbpxxw7xwx2gip4jjls6/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libffi-3.3-tzxzh4ht5w3cijtjmn4ughfr7llzrojx/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libelf-0.8.13-styifuipry54smq2ambvymmtasdpju75/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libedit-3.1-20210216-fcegcr5md4lua5ahuqhrh462cw7pmaul/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/hwloc-2.4.1-quyjzlvk6gc5dtq5i4keo5nchaceymrf/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libpciaccess-0.16-jml4hzwgnmtps5wh3wf5cy7bxxl6d673/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/ncurses-6.2-hznv2jsuev33sr4kgpyjdl6tbycbhmwt/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libxml2-2.9.10-lddvpplxexarse3lxzmalchfdk7gtz4r/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/zlib-1.2.11-uw42aiy6225mpwyeha5txnsde4pxaey2/lib/pkgconfig:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/xz-5.2.5-tnkfg2wegm2rnfqdwfnroclzam4qrxca/lib/pkgconfig:/p/software/jurecadc/stages/2020/software/zlib/1.2.11-GCCcore-9.3.0/lib/pkgconfig;
export CPATH=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/python-3.8.8-imhoumrie4wglmjsmlu3ovzmh2hkiayx/include/python3.8:/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/include:/p/software/jurecadc/stages/2020/software/zlib/1.2.11-GCCcore-9.3.0/include;
export SPACK_LOADED_HASHES=p3mbaxkch2s67xnnesfyybgp2uh5qs25:nnnpaka5o43bqefkytx4begoe4xjtiil:wk4ghswz5tx7az26ky2pxr6ieooslzg7:imhoumrie4wglmjsmlu3ovzmh2hkiayx:ongln3u6sqqymu7aspiqsi76qvvp4koq:7io6mp5ax2ddpaycp364quaonrpuffaj:s6szyx5khijyg6drni2yaharjcnmkau3:tsjaztspbf2z4aiyu56nwiwnfklo6nlx:5knyycilfqlyjjntb7yjdxmu327w7dkl:grguipxxnciycsmllwr6myjmpajxbkqz:ex3xe7rxzmjcnikxkqbzaxybw7vmzi2q:zsjwbhzfaqh3swtvasl7kdadrsp5my4m:ihgobdpfkpx3jbpxxw7xwx2gip4jjls6:tzxzh4ht5w3cijtjmn4ughfr7llzrojx:styifuipry54smq2ambvymmtasdpju75:fcegcr5md4lua5ahuqhrh462cw7pmaul:quyjzlvk6gc5dtq5i4keo5nchaceymrf:jml4hzwgnmtps5wh3wf5cy7bxxl6d673:tr6qhurpubqyiw2nomkplye7bf2yaprz:shxl54zbnjrlidbigrf3hsu3ywduplbl:2zusk3zerjuwltyzns7rlpqgdih6y4w5:x3qdbp3hbpb7lvhmwrpdjcuyo2ommzaw:hznv2jsuev33sr4kgpyjdl6tbycbhmwt:lddvpplxexarse3lxzmalchfdk7gtz4r:uw42aiy6225mpwyeha5txnsde4pxaey2:tnkfg2wegm2rnfqdwfnroclzam4qrxca:zmywoh3dwchbn67szay6bztarri4g3t2:72rw76qwcjufmomjqdldjilz2oeecjrx;
export CXX=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25/bin/clang++;
export PATH=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/z3-4.8.9-nnnpaka5o43bqefkytx4begoe4xjtiil/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/py-setuptools-50.3.2-wk4ghswz5tx7az26ky2pxr6ieooslzg7/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/python-3.8.8-imhoumrie4wglmjsmlu3ovzmh2hkiayx/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/util-linux-uuid-2.36-ongln3u6sqqymu7aspiqsi76qvvp4koq/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/sqlite-3.34.0-7io6mp5ax2ddpaycp364quaonrpuffaj/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/openssl-1.1.1k-s6szyx5khijyg6drni2yaharjcnmkau3/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gdbm-1.19-tsjaztspbf2z4aiyu56nwiwnfklo6nlx/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/readline-8.0-5knyycilfqlyjjntb7yjdxmu327w7dkl/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/expat-2.2.10-grguipxxnciycsmllwr6myjmpajxbkqz/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/swig-4.0.2-zsjwbhzfaqh3swtvasl7kdadrsp5my4m/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/pcre-8.44-ihgobdpfkpx3jbpxxw7xwx2gip4jjls6/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/hwloc-2.4.1-quyjzlvk6gc5dtq5i4keo5nchaceymrf/bin:/p/software/jurecadc/stages/2020/software/CUDA/11.0/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/binutils-2.36.1-shxl54zbnjrlidbigrf3hsu3ywduplbl/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gettext-0.21-2zusk3zerjuwltyzns7rlpqgdih6y4w5/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/tar-1.32-x3qdbp3hbpb7lvhmwrpdjcuyo2ommzaw/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/ncurses-6.2-hznv2jsuev33sr4kgpyjdl6tbycbhmwt/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libxml2-2.9.10-lddvpplxexarse3lxzmalchfdk7gtz4r/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/xz-5.2.5-tnkfg2wegm2rnfqdwfnroclzam4qrxca/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libiconv-1.16-zmywoh3dwchbn67szay6bztarri4g3t2/bin:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/bzip2-1.0.8-72rw76qwcjufmomjqdldjilz2oeecjrx/bin:/p/project/cjzam11/kitayama1/opt/nrn-master/bin:/p/project/cjzam11/kitayama1/projects/spack/bin:/p/project/cjzam11/kitayama1/projects/spack/bin:/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin:/p/software/jurecadc/stages/2020/software/GCCcore/9.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ddn/ime/bin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/p/software/jurecadc/bin;
export ACLOCAL_PATH=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/gettext-0.21-2zusk3zerjuwltyzns7rlpqgdih6y4w5/share/aclocal:/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/libxml2-2.9.10-lddvpplxexarse3lxzmalchfdk7gtz4r/share/aclocal;
export FC=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25/bin/flang;
export CC=/p/project/cjzam11/kitayama1/projects/spack/opt/spack/linux-centos8-zen2/gcc-10.2.0/llvm-main-p3mbaxkch2s67xnnesfyybgp2uh5qs25/bin/clang;
export CUDA_HOME=/p/software/jurecadc/stages/2020/software/CUDA/11.0;
         2220804 function calls (2107224 primitive calls) in 11.024 seconds

   Ordered by: internal time
   List reduced from 1522 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    10874    7.723    0.001    7.723    0.001 {built-in method posix.stat}
        2    1.338    0.669    1.338    0.669 {method 'read' of '_io.BufferedReader' objects}
        6    0.254    0.042    0.254    0.042 {built-in method posix.open}
       18    0.095    0.005    0.095    0.005 {method 'readline' of '_io.BufferedReader' objects}
       82    0.092    0.001    0.118    0.001 <frozen importlib._bootstrap_external>:830(get_data)
      448    0.083    0.000    0.083    0.000 {method 'read' of '_io.TextIOWrapper' objects}
     8783    0.073    0.000    0.082    0.000 /usr/lib64/python3.6/shlex.py:129(read_token)
 49478/38    0.049    0.000    0.118    0.003 /usr/lib64/python3.6/copy.py:132(deepcopy)
        6    0.048    0.008    0.048    0.008 {method 'poll' of 'select.poll' objects}
173074/172973    0.031    0.000    0.034    0.000 {built-in method builtins.isinstance}
       38    0.030    0.001    0.031    0.001 {built-in method io.open}
     8783    0.027    0.000    0.115    0.000 /usr/lib64/python3.6/shlex.py:294(__next__)
    15392    0.027    0.000    0.042    0.000 /usr/lib64/python3.6/posixpath.py:75(join)
       82    0.026    0.000    0.026    0.000 {method 'read' of '_io.FileIO' objects}
     4441    0.024    0.000    0.069    0.000 /usr/lib64/python3.6/re.py:362(scan)
     6354    0.024    0.000    0.055    0.000 /p/project/cjzam11/kitayama1/projects/spack/lib/spack/spack/version.py:102(__init__)
     6255    0.022    0.000    0.031    0.000 /p/project/cjzam11/kitayama1/projects/spack/lib/spack/spack/version.py:383(__init__)
        7    0.021    0.003    0.024    0.003 /usr/lib64/python3.6/json/decoder.py:345(raw_decode)
25632/5457    0.020    0.000    0.057    0.000 /p/project/cjzam11/kitayama1/projects/spack/lib/spack/spack/version.py:89(coercing_method)
8660/7668    0.017    0.000    0.036    0.000 /p/project/cjzam11/kitayama1/projects/spack/lib/spack/spack/version.py:600(add)
```
Does Spack redundantly do file operations?
@haampie , @tgamblin , @becker33 ?
This is a sore point while using Spack on GPFS.

Can you try to figure out where the 10874 stat calls come from? Seems like it's redundant if the output is just a few exported env variables...
@haampie how do I do that?
A very rudimentary way is to just comment out a few lines in `lib/spack/spack/cmd/load.py` or `lib/spack/spack/user_environment.py` and run the profiler again
Let me try that and report back here over the weekend; today's Friday here and am kind of busy with other things.
Those are from `FastPackageChecker` in `repo.py`.  It's used to figure out when we need to reindex the repo.  In particular, the call to `is_virtual` in there triggers it; it has a `use_index` option to use the index generated when `FastPackageChecker` discovers some package is out of date.

There is a tricky interaction between the various package indices (e.g., of virtuals, patch SHA's, etc.) and various commands.  I think in this case it's clear that `spack load` only really needs info about certain packages, and you probably do not need to trigger repo indexing process.  I haven't had a chance to look through the code, but really `spack load` shouldn't need to consult the package index too much at all -- it just executes code in the `package.py` file for each package in the loaded `spec`.  I think you could get away with not reindexing on `load` -- I just haven't looked at how easy it is to disentangle that from other parts of `repo.py`.
Also, if you want a better sense of where performance issues come from, use `pyinstrument`.  You should be able to install `py-pyinstrument` in a spack environment, then use its python to run spack.  If you do that, you can run something like this:

```console
python3 -m pyinstrument --show-all --hide xxx $(which spack) load --sh <thing you want to load>
```

`--show-all --hide xxx` will ensure that all frames are shown -- you may not need them but I sometimes find them useful.

That command will give you a call path profile and you can see the parts taking up the most time.  I'd be interested to see what it gives you.

Example output on my macbook here (this won't come close to showing your GPFS problem):

```
(spackle):spack> python3 -m pyinstrument $(which spack) load --sh zlib/f7owbzf
export DYLD_FALLBACK_LIBRARY_PATH=/Users/gamblin2/Workspace/spack/opt/spack/darwin-catalina-skylake/apple-clang-12.0.0/zlib-1.2.11-f7owbzfu4zsr6yn7x7hljransoof6ach/lib;
export PKG_CONFIG_PATH=/Users/gamblin2/Workspace/spack/opt/spack/darwin-catalina-skylake/apple-clang-12.0.0/zlib-1.2.11-f7owbzfu4zsr6yn7x7hljransoof6ach/lib/pkgconfig;
export MANPATH=/Users/gamblin2/Workspace/spack/opt/spack/darwin-catalina-skylake/apple-clang-12.0.0/zlib-1.2.11-f7owbzfu4zsr6yn7x7hljransoof6ach/share/man;
export CMAKE_PREFIX_PATH=/Users/gamblin2/Workspace/spack/opt/spack/darwin-catalina-skylake/apple-clang-12.0.0/zlib-1.2.11-f7owbzfu4zsr6yn7x7hljransoof6ach;
export SPACK_LOADED_HASHES=f7owbzfu4zsr6yn7x7hljransoof6ach;

  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:13:05  Samples:  443
 /_//_/// /_\ / //_// / //_'/ //     Duration: 0.518     CPU time: 0.485
/   _/                      v3.3.0

Program: /Users/gamblin2/src/spack/bin/spack load --sh zlib/f7owbzf

0.517 <module>  spack:23
├─ 0.358 main  main.py:686
│     [873 frames hidden]  main, cmd, spec, parse, repo, posixpa...
│        0.096 _create_new_cache  repo.py:156
│        ├─ 0.043 stat  <built-in>:0
│        │  0.010 match  re.py:172
│        │  └─ 0.006 Pattern.match  <built-in>:0
│        └─ 0.006 listdir  <built-in>:0
│        0.014 load_source  util/imp/importlib_importer.py:32
│        └─ 0.013 <module>  ../../var/spack/repos/builtin/packages/zlib/package.py:1
│           └─ 0.013 <module>  pkgkit.py:10
│                 [19 frames hidden]  pkgkit, build_systems, xml, re, sre_c...
│        0.013 communicate  subprocess.py:920
│        └─ 0.012 BufferedReader.read  <built-in>:0
│        0.006 _execute_child  subprocess.py:1412
│        └─ 0.006 fork_exec  <built-in>:0
│        0.014 islink  posixpath.py:168
│        └─ 0.014 lstat  <built-in>:0
│        0.020 islink  posixpath.py:168
│        └─ 0.019 lstat  <built-in>:0
│        0.005 read_or_stop  tokenize.py:374
│        └─ 0.005 BufferedReader.readline  <built-in>:0
└─ 0.159 <module>  main.py:10
      [264 frames hidden]  main, architecture, compiler, spec, r...
         0.016 <module>  urllib/request.py:68
         └─ 0.005 create_dynamic  <built-in>:0
         0.012 <module>  llnl/util/tty/__init__.py:6
         └─ 0.006 create_dynamic  <built-in>:0

To view this report with different options, run:
    pyinstrument --load-prev 2021-04-15T17-13-05 [options]

```
@tgamblin thanks for teaching us the module!

```

  _     ._   __/__   _ _  _  _ _/_   Recorded: 06:24:28  Samples:  6887
 /_//_/// /_\ / //_// / //_'/ //     Duration: 11.328    CPU time: 1.634
/   _/                      v3.1.0

Program: /p/project/cjzam11/kitayama1/projects/spack/bin/spack load --sh llvm/lvf

11.327 <module>  spack:12
├─ 10.756 main  spack/main.py:686
│     [1658 frames hidden]  spack, posixpath, re, llnl, genericpa...
│        6.781 _create_new_cache  spack/repo.py:156
│        └─ 6.759 [self]
│        0.129 load_source  spack/util/imp/importlib_importer.py:32
│        └─ 0.127 <module>  ../var/spack/repos/builtin/packages/llvm/package.py:1
│        3.200 environment_modifications_for_spec  spack/user_environment.py:68
│        ├─ 1.422 setup_run_environment  ../var/spack/repos/builtin/packages/python/package.py:833
│        │  └─ 1.367 __get__  spack/spec.py:901
│        │        [2 frames hidden]  spack
│        │           1.367 <lambda>  spack/spec.py:940
│        │           └─ 1.367 headers  ../var/spack/repos/builtin/packages/python/package.py:798
│        │              └─ 1.347 get_config_h_filename  ../var/spack/repos/builtin/packages/python/package.py:684
│        │                 └─ 1.345 __call__  spack/util/executable.py:76
│        │                       [11 frames hidden]  spack, subprocess, abc, os, _collecti...
│        └─ 0.368 modifications_from_dependencies  spack/build_environment.py:861
│           ├─ 0.193 setup_dependent_package  ../var/spack/repos/builtin/packages/python/package.py:935
│           │  └─ 0.183 home  ../var/spack/repos/builtin/packages/python/package.py:737
│           │     └─ 0.183 get_config_var  ../var/spack/repos/builtin/packages/python/package.py:675
│           │        └─ 0.182 __call__  spack/util/executable.py:76
│           │              [8 frames hidden]  spack, subprocess, os
└─ 0.566 <module>  spack/main.py:6
      [318 frames hidden]  spack, six, urllib, http, email, re, ...
```
@ikitayama I'm kinda curious if https://github.com/haampie/spack-batteries-included improves performance in any way, since it should have better caching behavior on a shared filesystem.

```console
wget https://github.com/haampie/spack-batteries-included/releases/download/develop/spack.x
chmod +x spack.x
# you may need to set your config in ~/.spack/config.yaml so that the config:install_prefix:root value is correct (or maybe copy spack config blame config values over to ~/.spack/config.yaml).
./spack.x load --sh llvm/lvf
```
Assume I'm testing your script correctly, on JURECA-DC, I am afraid I don't
feel the speed up at all.

On Mon, Apr 19, 2021 at 4:52 PM Harmen Stoppels ***@***.***>
wrote:

> @ikitayama <https://github.com/ikitayama> I'm kinda curious if
> https://github.com/haampie/spack-batteries-included improves performance
> in any way, since it should have better caching behavior on a shared
> filesystem.
>
> wget https://github.com/haampie/spack-batteries-included/releases/download/develop/spack.xchmod +x spack.x
> # you may need to set your config in ~/.spack/config.yaml so that the config:install_prefix:root value is correct (or maybe copy spack config blame config values over to ~/.spack/config.yaml)../spack.x load --sh llvm/lvf
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/spack/spack/issues/19871#issuecomment-822255373>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHLBZ3ZAMGDSPITHX6LCGDTJPOMZANCNFSM4TSYO3FQ>
> .
>

Hm okay, do you have more than just the builtin repo? Basically the `_create_new_cache` should be a bit faster in terms of filesystem access. Also, are you on the Spack Slack workspace? That'd be a bit easier to discuss
No, I don't think so at least, locally I never developed specs by myself.

On Mon, Apr 19, 2021 at 5:25 PM Harmen Stoppels
***@***.***> wrote:
>
> Hm okay, do you have more than just the builtin repo?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

@haampie are you on GPFS or Lustre? Looks like JSC's GPFS has been unstable for some time; that's I suspect, one reason Spack can't operate at full speed. 
The GPFS which is connected to JURECA-DC at JSC seems unstable these days, at least to me, so here's today's measurement:

```
  _     ._   __/__   _ _  _  _ _/_   Recorded: 06:08:46  Samples:  6441
 /_//_/// /_\ / //_// / //_'/ //     Duration: 9.236     CPU time: 1.473
/   _/                      v3.1.0

Program: /p/scratch/cjzam11/kitayama1/projects/spack/bin/spack load --sh llvm/dmh

9.235 <module>  spack:12
├─ 8.575 main  spack/main.py:686
│     [1515 frames hidden]  spack, posixpath, re, llnl, genericpa...
│        6.745 _create_new_cache  spack/repo.py:156
│        └─ 6.689 [self]
│        0.142 load_source  spack/util/imp/importlib_importer.py:32
│        └─ 0.139 <module>  ../var/spack/repos/builtin/packages/llvm/package.py:1
│           └─ 0.096 <module>  spack/pkgkit.py:7
│                 [14 frames hidden]  spack, xml
│        1.276 environment_modifications_for_spec  spack/user_environment.py:68
│        └─ 0.173 setup_run_environment  ../var/spack/repos/builtin/packages/python/package.py:833
│           └─ 0.115 __get__  spack/spec.py:901
│                 [2 frames hidden]  spack
│                    0.115 <lambda>  spack/spec.py:940
│                    └─ 0.115 headers  ../var/spack/repos/builtin/packages/python/package.py:798
│                       └─ 0.112 get_config_h_filename  ../var/spack/repos/builtin/packages/python/package.py:684
│                          └─ 0.111 __call__  spack/util/executable.py:76
│                                [8 frames hidden]  spack, subprocess, os
└─ 0.621 <module>  spack/main.py:6
      [303 frames hidden]  spack, six, urllib, http, email, re, ...
```
Is loading time constant for any specs?
@alalazo , I'd like to investigate this issue with your help; currently, I have access to the local BeeGFS system.

Resolving patch:
diff --git a/lib/spack/spack/repo.py b/lib/spack/spack/repo.py
--- a/lib/spack/spack/repo.py
+++ b/lib/spack/spack/repo.py
@@ -918,8 +918,12 @@ def _read_config(self):
     @autospec
     def get(self, spec):
         """Returns the package associated with the supplied spec."""
-        if not self.exists(spec.name):
-            raise UnknownPackageError(spec.name)
+        # NOTE: we only check whether the package is None here, not whether it
+        # actually exists, because we have to load it anyway, and that ends up
+        # checking for existence. We avoid constructing FastPackageChecker,
+        # which will stat all packages.
+        if spec.name is None:
+            raise UnknownPackageError(None, self)
 
         if spec.namespace and spec.namespace != self.namespace:
             raise UnknownPackageError(spec.name, self.namespace)
@@ -1064,7 +1068,16 @@ def all_package_classes(self):
 
     def exists(self, pkg_name):
         """Whether a package with the supplied name exists."""
-        return pkg_name in self._pkg_checker
+        if pkg_name is None:
+            return False
+
+        # if the FastPackageChecker is already constructed, use it
+        if self._fast_package_checker:
+            return pkg_name in self._pkg_checker
+
+        # if not, check for the package.py file
+        path = self.filename_for_package_name(pkg_name)
+        return os.path.exists(path)
 
     def last_mtime(self):
         """Time a package file in this repo was last updated."""
@@ -1331,7 +1344,7 @@ def __init__(self, name, repo=None):
         long_msg = None
         if name:
             if repo:
-                msg = "Package '{0}' not found in repository '{1}'"
+                msg = "Package '{0}' not found in repository '{1.root}'"
                 msg = msg.format(name, repo)
             else:
                 msg = "Package '{0}' not found.".format(name)
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -3193,25 +3193,23 @@ def satisfies(self, other, deps=True, strict=False, strict_deps=False):
         if other.concrete:
             return self.concrete and self.dag_hash() == other.dag_hash()
 
-        # A concrete provider can satisfy a virtual dependency.
-        if not self.virtual and other.virtual:
-            try:
-                pkg = spack.repo.get(self.fullname)
-            except spack.repo.UnknownEntityError:
-                # If we can't get package info on this spec, don't treat
-                # it as a provider of this vdep.
-                return False
-
-            if pkg.provides(other.name):
-                for provided, when_specs in pkg.provided.items():
-                    if any(self.satisfies(when_spec, deps=False, strict=strict)
-                           for when_spec in when_specs):
-                        if provided.satisfies(other):
-                            return True
-            return False
-
-        # Otherwise, first thing we care about is whether the name matches
+        # If the names are different, we need to consider virtuals
         if self.name != other.name and self.name and other.name:
+            # A concrete provider can satisfy a virtual dependency.
+            if not self.virtual and other.virtual:
+                try:
+                    pkg = spack.repo.get(self.fullname)
+                except spack.repo.UnknownEntityError:
+                    # If we can't get package info on this spec, don't treat
+                    # it as a provider of this vdep.
+                    return False
+
+                if pkg.provides(other.name):
+                    for provided, when_specs in pkg.provided.items():
+                        if any(self.satisfies(when, deps=False, strict=strict)
+                               for when in when_specs):
+                            if provided.satisfies(other):
+                                return True
             return False
 
         # namespaces either match, or other doesn't require one.
@@ -4639,7 +4637,8 @@ def spec(self, name):
                     break
 
             elif self.accept(HASH):
-                # Get spec by hash and confirm it matches what we already have
+                # Get spec by hash and confirm it matches any constraints we
+                # already read in
                 hash_spec = self.spec_by_hash()
                 if hash_spec.satisfies(spec):
                     spec._dup(hash_spec)


Test patch:
diff --git a/lib/spack/spack/test/cmd/pkg.py b/lib/spack/spack/test/cmd/pkg.py
--- a/lib/spack/spack/test/cmd/pkg.py
+++ b/lib/spack/spack/test/cmd/pkg.py
@@ -129,7 +129,8 @@ def test_pkg_add(mock_pkg_git_repo):
         finally:
             shutil.rmtree('pkg-e')
             # Removing a package mid-run disrupts Spack's caching
-            spack.repo.path.repos[0]._fast_package_checker.invalidate()
+            if spack.repo.path.repos[0]._fast_package_checker:
+                spack.repo.path.repos[0]._fast_package_checker.invalidate()
 
     with pytest.raises(spack.main.SpackCommandError):
         pkg('add', 'does-not-exist')