Repository: spack/spack

Issue:
Spack Environments are painfully slow
I've been using Spack Environments for a while now, and so far my biggest complaint is the performance. On an NFS cluster in particular, it can take upwards of 15 minutes just to concretize an environment. Things are particularly bad when the environment is partially installed, as most of the time is spent updating the environment, not actually concretizing anything. I've even noticed that commands like `spack edit` or `spack help` will take orders of magnitude longer if I'm running them from a directory containing a `spack.yaml`. If I run `spack -d concretize`, I notice that we are acquiring and releasing locks on Spack's database thousands of times. Is there any way we can hold on to the lock until another process requests it so we don't need all this overhead?

@tgamblin 


Discussion:
Not sure if this is directly relevant, but very recently it seems the time it takes for spack to activate an environment has became considerably slow.
@Sinan81 I'm guessing this is related to #13249. You can try the commits before and after if you want to confirm and give some benchmarks.
> @Sinan81 I'm guessing this is related to #13249. You can try the commits before and after if you want to confirm and give some benchmarks.

I was also guessing  this PR could be the reason for the slow down
I ran some quick benchmarks on Blue Waters. From my home directory:
```console
$ time spack help
...
real	0m22.465s
user	0m2.236s
sys	0m1.344s
```
From a directory containing a `spack.yaml` file:
```console
$ time spack help
...
real	2m58.860s
user	2m2.912s
sys	0m15.233s
```
At these speeds, Spack is practically useless. I can't even `spack edit` a file unless I want to wait around for 3 minutes.
On my personal laptop, the difference is even greater. From my home directory:
```console
$ time spack help
...
real	0m4.016s
user	0m0.738s
sys	0m0.379s
```
From a directory containing a smaller `spack.yaml` (uninstalled environment):
```console
$ time spack help
...
real	0m9.092s
user	0m7.952s
sys	0m0.434s
```
From a directory containing a larger `spack.yaml` (installed environment):
```console
$ time spack help
...
real	2m7.786s
user	2m1.647s
sys	0m2.491s
```
That's a 2x slow down for a small `spack.yaml`, and a 30x slow down for a large `spack.yaml` that has been installed.
@adamjstewart Can you share your `spack.yaml` files? I see a minimal slow down on my laptop with the mock environments I tried so far. Without environments:
```console
$ time spack help &> /dev/null

real    0m0,618s
user    0m0,532s
sys     0m0,068s
```
With this `spack.yaml`:
```yaml
spack:
  specs:
  - mvapich2 fabrics=mrail
  - hdf5+mpi
  - gromacs
  - python
  view: false
  concretization: together
```
I get instead:
```console
$ spack concretize
[ ... ]

$ time spack help &> /dev/null

real    0m0,762s
user    0m0,704s
sys     0m0,041s
```
so it seems there is a slow down but not as dramatic as the one reported here. I am trying to figure out if I miss a trigger for it to be much worse.
For sure having the environment installed makes things worse. With the environment posted above:
```console
$ spack install
[ ... ]

$ time spack help &> /dev/null

real    0m2,362s
user    0m1,235s
sys     0m0,130s
```
even though subsequent calls seem to be as fast as before:
```console
$ time spack help &> /dev/null

real    0m0,812s
user    0m0,729s
sys     0m0,065s
```
@adamjstewart I've been trying to reproduce the slow down, but I am unable right now. Waiting to try your `spack.yaml`.
@adamjstewart Nevermind. I passed to #13529 and with that `spack.yaml` I see **huge** slow downs. I suspect we do a lot of work that is not needed when we have packages that can be extended.
@adamjstewart I'll try to debug this, the issue seems to be within activation and deactivation of environments. Once the environment is activated debug info shows that the time to execute commands is pretty much the same as outside an environment.
@alalazo @adamjstewart: just woke up or I’d do it myself, but try a quick fix by putting a read transaction around the loops in `add_default_view_to_spec` (as `environment_modifications_for_spec` calls `spec.prefix`).  This is similar to what I did in  #12686.
@tgamblin Doing it right now.
Could this https://github.com/spack/spack/issues/12779  be related?
So, without that protection:
```console
$ time spack env activate -d --sh .
[ ... ]
real    0m27,346s
user    0m26,379s
sys     0m0,553s
```
and with the context manager:
```console
$ time spack env activate -d --sh .
[ ... ]
real    0m11,238s
user    0m10,942s
sys     0m0,274s
```
It seems that's the right direction, but probably we are still missing some other place. I'll keep looking.
The other thing that may be a problem is that it’s just iterating over package files. If the env is *just* being activated for a local spack.yaml, we don’t need all the env modifications.  They shouldn’t affect the build or spack.  If we’re activating with `spack env activate`, then we do want them for the user’s environment.

So you could just avoid that loop through the packages altogether when activating for a local spack.yaml. I think environment.get can make that decision.
@tgamblin Just to be faster, since I'm not too familiar with that part of the code. Is there any reason why I don't get profile data when using `spack -p`:
```console
$ spack -p env activate --sh -d /home/culpo/tmp/spack-environments/issues/13529 
export SPACK_ENV=/home/culpo/tmp/spack-environments/issues/13529;
alias despacktivate='spack env deactivate';
export PKG_CONFIG_PATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib/pkgconfig:/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib64/pkgconfig;
export PATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/bin:/home/culpo/PycharmProjects/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-9.0.1/environment-modules-4.3.1-n55rsnwwhs6nh2wrjhhix3fyh7trcfww/bin:/home/culpo/PycharmProjects/spack/bin:/home/culpo/development/venv/spack/py37/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/culpo/go/bin:/opt/tsetup.1.2.17/Telegram;
export CMAKE_PREFIX_PATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view;
export LIBRARY_PATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib:/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib64;
export CPATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/include;
export ACLOCAL_PATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/share/aclocal;
export DYLD_LIBRARY_PATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib:/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib64;
export MANPATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/share/man:/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/man;
export LD_LIBRARY_PATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib:/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib64;
export PYTHONPATH=/home/culpo/tmp/spack-environments/issues/13529/.spack-env/view/lib/python3.7/site-packages;
```
? This happens both within the directory with the `spack.yaml` and outside. Within the directory the times are doubled, which makes me think I pay for implicit activation on top of the one required by the command.
Not sure. I usually find `python -m pyinstrument $(which spack) ...` useful....
Thanks @tgamblin will try that. In the meantime confirmed that the other part severely affecting performance is:

https://github.com/spack/spack/blob/903b0071cf9299f0d4e950a41a4ec9d95df9a43d/lib/spack/spack/environment.py#L1111-L1118

Resolving patch:
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -422,6 +422,11 @@ def set_build_environment_variables(pkg, env, dirty):
 
 def _set_variables_for_single_module(pkg, module):
     """Helper function to set module variables for single module."""
+    # Put a marker on this module so that it won't execute the body of this
+    # function again, since it is not needed
+    marker = '_set_run_already_called'
+    if getattr(module, marker, False):
+        return
 
     jobs = spack.config.get('config:build_jobs') if pkg.parallel else 1
     jobs = min(jobs, multiprocessing.cpu_count())
@@ -489,6 +494,10 @@ def static_to_shared_library(static_lib, shared_lib=None, **kwargs):
 
     m.static_to_shared_library = static_to_shared_library
 
+    # Put a marker on this module so that it won't execute the body of this
+    # function again, since it is not needed
+    setattr(m, marker, True)
+
 
 def set_module_variables_for_package(pkg):
     """Populate the module scope of install() with some useful functions.
diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py
--- a/lib/spack/spack/environment.py
+++ b/lib/spack/spack/environment.py
@@ -159,7 +159,8 @@ def activate(
             cmds += 'export PS1="%s ${PS1}";\n' % prompt
 
     if add_view and default_view_name in env.views:
-        cmds += env.add_default_view_to_shell(shell)
+        with spack.store.db.read_transaction():
+            cmds += env.add_default_view_to_shell(shell)
 
     return cmds
 
@@ -207,7 +208,8 @@ def deactivate(shell='sh'):
         cmds += 'fi;\n'
 
     if default_view_name in _active_environment.views:
-        cmds += _active_environment.rm_default_view_from_shell(shell)
+        with spack.store.db.read_transaction():
+            cmds += _active_environment.rm_default_view_from_shell(shell)
 
     tty.debug("Deactivated environmennt '%s'" % _active_environment.name)
     _active_environment = None
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -163,6 +163,9 @@ class Python(AutotoolsPackage):
     _DISTUTIL_CACHE_FILENAME = 'sysconfig.json'
     _distutil_vars = None
 
+    # Used to cache home locations, since computing them might be expensive
+    _homes = {}
+
     # An in-source build with --enable-optimizations fails for python@3.X
     build_directory = 'spack-build'
 
@@ -622,8 +625,11 @@ def home(self):
         ``packages.yaml`` unknowingly. Query the python executable to
         determine exactly where it is installed."""
 
-        prefix = self.get_config_var('prefix')
-        return Prefix(prefix)
+        dag_hash = self.spec.dag_hash()
+        if dag_hash not in self._homes:
+            prefix = self.get_config_var('prefix')
+            self._homes[dag_hash] = Prefix(prefix)
+        return self._homes[dag_hash]
 
     @property
     def libs(self):


Test patch: