Repository: spack/spack

Issue:
Environment don't account for custom install tree in spack.yaml
### Steps to reproduce

A custom store in an environment doesn't seem to be taken into account during concretization:
```console
root@f3da03c21c0d:/# mkdir -p spack && cd spack
root@f3da03c21c0d:/spack# git clone https://github.com/spack/spack.git
[ ... ]
root@f3da03c21c0d:/spack# . spack/share/spack/setup-env.sh 
root@f3da03c21c0d:/spack# mkdir -p /spack-env && cd /spack-env
root@f3da03c21c0d:/spack-env# cat spack.yaml 
spack:
  specs:
  - zlib@1.2:1.2.99
  config:
    install_tree:
      root: ./store
  view: false

root@f3da03c21c0d:/spack-env# spack -e . concretize -f
==> Bootstrapping clingo from pre-built binaries
==> Concretized zlib@1.2:1.2.99
 -   ech7q6s  zlib@1.2.11%gcc@9.3.0+optimize+pic+shared arch=linux-ubuntu20.04-icelake

root@f3da03c21c0d:/spack-env# spack -e . install      
==> Installing environment /spack-env
==> Installing zlib-1.2.11-ech7q6sqptumrsjdsvureubhkov2dm23
==> No binary for zlib-1.2.11-ech7q6sqptumrsjdsvureubhkov2dm23 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/c3/c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1.tar.gz
==> No patches needed for zlib
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.11-ech7q6sqptumrsjdsvureubhkov2dm23
  Fetch: 0.22s.  Build: 1.81s.  Total: 2.03s.
[+] /spack-env/store/linux-ubuntu20.04-icelake/gcc-9.3.0/zlib-1.2.11-ech7q6sqptumrsjdsvureubhkov2dm23
root@f3da03c21c0d:/spack-env# spack -e . concretize -f
==> Concretized zlib@1.2:1.2.99
 -   ech7q6s  zlib@1.2.11%gcc@9.3.0+optimize+pic+shared arch=linux-ubuntu20.04-icelake
```
The output above shows a `-` symbol even after installation. Trying instead:
```console
root@f3da03c21c0d:/spack-env# cd /spack/spack && git checkout 713bbdbe7c1c2078abdc09bf515b3f934e87d741 && cd -
root@f3da03c21c0d:/spack-env# spack -e . concretize -f
==> Concretized zlib@1.2:1.2.99
[+]  ech7q6s  zlib@1.2.11%gcc@9.3.0+optimize+pic+shared arch=linux-ubuntu20.04-icelake
```
shows that zlib is already installed in the environment. 

Originally I experienced this issue with an already existing environment that was installing specs in the default store (instead of the custom one) after a git pull. I can't reproduce anymore the installation issue after having removed all the caches and `~/.spack` completely.

### Error message

There's no error message, but Spack ~installs software in the wrong prefix.~ doesn't show concretized software as installed when it instead is. Bisecting the issue points at #25608 (the first bad commit is d998ea1bd41b4cf7f47caa734b7c7df86178ef0c)

### Information on your system

* **Spack:** 0.16.3-4720-199bd9f8f8
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-icelake
* **Concretizer:** clingo

[Dockerfile](https://github.com/spack/spack/files/7292873/Dockerfile.txt)


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
@haampie Can you reproduce this?
can't reproduce?
For the record, trying to triage this in DM with @haampie The actual environment is more complex and was already installed. I'll be posting a reproducer in a container if I find it.
Applying the following:
```diff
diff --git a/lib/spack/spack/environment/environment.py b/lib/spack/spack/environment/environment.py
index 88dd545398..919d5ab055 100644
--- a/lib/spack/spack/environment/environment.py
+++ b/lib/spack/spack/environment/environment.py
@@ -1908,7 +1908,7 @@ def _tree_to_display(spec):
 
     for user_spec, concrete_spec in concretized_specs:
         tty.msg('Concretized {0}'.format(user_spec))
-        sys.stdout.write(_tree_to_display(concrete_spec))
+        sys.stdout.write(concrete_spec.prefix)
         print('')
 

```
within the environment in the description, this is the output:
```console
root@f3da03c21c0d:/spack-env# spack -e . concretize -f
==> Concretized zlib@1.2:1.2.99
/spack/spack/opt/spack/linux-ubuntu20.04-icelake/gcc-9.3.0/zlib-1.2.11-ech7q6sqptumrsjdsvureubhkov2dm23
```
which points to a wrong prefix (should be `/spack-env/store/...`). Doing a similar thing in the commit posted above gives instead:
```console
root@f3da03c21c0d:/spack-env# spack -e . concretize -f
==> Concretized zlib@1.2:1.2.99
/spack-env/store/linux-ubuntu20.04-icelake/gcc-9.3.0/zlib-1.2.11-ech7q6sqptumrsjdsvureubhkov2dm23
```
which is the expected value.

Resolving patch:
diff --git a/lib/spack/spack/environment/environment.py b/lib/spack/spack/environment/environment.py
--- a/lib/spack/spack/environment/environment.py
+++ b/lib/spack/spack/environment/environment.py
@@ -126,7 +126,14 @@ def activate(env, use_env_repo=False):
     if not isinstance(env, Environment):
         raise TypeError("`env` should be of type {0}".format(Environment.__name__))
 
+    # Check if we need to reinitialize the store due to pushing the configuration
+    # below.
+    store_before_pushing = spack.config.get('config:install_tree')
     prepare_config_scope(env)
+    store_after_pushing = spack.config.get('config:install_tree')
+    if store_before_pushing != store_after_pushing:
+        # Hack to store the state of the store before activation
+        env.store_token = spack.store.reinitialize()
 
     if use_env_repo:
         spack.repo.path.put_first(env.repo)
@@ -144,6 +151,11 @@ def deactivate():
     if not _active_environment:
         return
 
+    # If we attached a store token on activation, restore the previous state
+    # and consume the token
+    if hasattr(_active_environment, 'store_token'):
+        spack.store.restore(_active_environment.store_token)
+        delattr(_active_environment, 'store_token')
     deactivate_config_scope(_active_environment)
 
     # use _repo so we only remove if a repo was actually constructed
diff --git a/lib/spack/spack/store.py b/lib/spack/spack/store.py
--- a/lib/spack/spack/store.py
+++ b/lib/spack/spack/store.py
@@ -236,17 +236,29 @@ def _store_layout():
 
 
 def reinitialize():
-    """Restore globals to the same state they would have at start-up"""
+    """Restore globals to the same state they would have at start-up. Return a token
+    containing the state of the store before reinitialization.
+    """
     global store
     global root, unpadded_root, db, layout
 
-    store = llnl.util.lang.Singleton(_store)
+    token = store, root, unpadded_root, db, layout
 
+    store = llnl.util.lang.Singleton(_store)
     root = llnl.util.lang.LazyReference(_store_root)
     unpadded_root = llnl.util.lang.LazyReference(_store_unpadded_root)
     db = llnl.util.lang.LazyReference(_store_db)
     layout = llnl.util.lang.LazyReference(_store_layout)
 
+    return token
+
+
+def restore(token):
+    """Restore the environment from a token returned by reinitialize"""
+    global store
+    global root, unpadded_root, db, layout
+    store, root, unpadded_root, db, layout = token
+
 
 def retrieve_upstream_dbs():
     other_spack_instances = spack.config.get('upstreams', {})


Test patch:
diff --git a/lib/spack/spack/test/bootstrap.py b/lib/spack/spack/test/bootstrap.py
--- a/lib/spack/spack/test/bootstrap.py
+++ b/lib/spack/spack/test/bootstrap.py
@@ -118,3 +118,25 @@ def test_config_yaml_is_preserved_during_bootstrap(mutable_config):
     with spack.bootstrap.ensure_bootstrap_configuration():
         assert spack.config.get('config:test_stage') == expected_dir
     assert spack.config.get('config:test_stage') == expected_dir
+
+
+@pytest.mark.regression('26548')
+def test_custom_store_in_environment(mutable_config, tmpdir):
+    # Test that the custom store in an environment is taken into account
+    # during bootstrapping
+    spack_yaml = tmpdir.join('spack.yaml')
+    spack_yaml.write("""
+spack:
+  specs:
+  - libelf
+  config:
+    install_tree:
+      root: /tmp/store
+""")
+    with spack.environment.Environment(str(tmpdir)):
+        assert spack.environment.active_environment()
+        assert spack.config.get('config:install_tree:root') == '/tmp/store'
+        # Don't trigger evaluation here
+        with spack.bootstrap.ensure_bootstrap_configuration():
+            pass
+        assert str(spack.store.root) == '/tmp/store'
diff --git a/lib/spack/spack/test/cmd/env.py b/lib/spack/spack/test/cmd/env.py
--- a/lib/spack/spack/test/cmd/env.py
+++ b/lib/spack/spack/test/cmd/env.py
@@ -2663,3 +2663,21 @@ def test_activation_and_deactiviation_ambiguities(method, env, no_env, env_dir,
         method(args)
     _, err = capsys.readouterr()
     assert 'is ambiguous' in err
+
+
+@pytest.mark.regression('26548')
+def test_custom_store_in_environment(mutable_config, tmpdir):
+    spack_yaml = tmpdir.join('spack.yaml')
+    spack_yaml.write("""
+spack:
+  specs:
+  - libelf
+  config:
+    install_tree:
+      root: /tmp/store
+""")
+    current_store_root = str(spack.store.root)
+    assert str(current_store_root) != '/tmp/store'
+    with spack.environment.Environment(str(tmpdir)):
+        assert str(spack.store.root) == '/tmp/store'
+    assert str(spack.store.root) == current_store_root