Repository: spack/spack

Issue:
Clingo Concretizer Doesn't Respect Custom Install Tree
<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `spack find` to list all the installed packages and ..." -->

I ran spack containerize with the following spack environment  file and when building the resulting Dockerfile spack continued to install software into `/opt/spack/var/spack` instead of the set install path which results in an empty container being produced. Setting the concretizer to  `original` fixes the issue. Cc'ing @vsoch.

```yaml
spack:
  specs: wget
  view: true
  config:
    concretizer: clingo
  container:
    os_packages:
      build:
        - python3-dev
    strip: false
```

Dockerfile
```Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:latest as builder

# Install OS packages needed to build the software
RUN apt-get -yqq update && apt-get -yqq upgrade \
 && apt-get -yqq install python3-dev \
 && rm -rf /var/lib/apt/lists/*

# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo "spack:" \
&&   echo "  specs:" \
&&   echo "  - wget" \
&&   echo "  view: /opt/view" \
&&   echo "  config:" \
&&   echo "    concretizer: clingo" \
&&   echo "    install_tree: /opt/software" \
&&   echo "  concretization: together") > /opt/spack-environment/spack.yaml

# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y

# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh


# Bare OS image to run the installed executables
FROM ubuntu:18.04

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh



ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l", "-c"]
```

### Steps to reproduce the issue

```console
$ spack containerize
$ docker run build -t wget .
```

### Information on your system

<!-- Please include the output of `spack debug report` -->
* **Spack:** 0.16.1
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-skylake
* **Concretizer:** clingo
<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
I need to double check, but it's possible this was solved by #22559 
It doesn't seem to be solved by #22559 I'll investigate this further and report here the issues I found.
I could reproduce even without the containerize command:
```yaml
spack:
  specs:
  - openjpeg
  config:
    concretizer: clingo
    install_tree:
      root: ./store
```
The bug is related to the bootstrapping procedure, and most likely to the swapping of some part of the configuration. Starting with a python interpreter that can already import `clingo` doesn't trigger the bug.
The issue is with temporary caches and lazily evaluated values. Currently when we bootstrap we call this method:

https://github.com/spack/spack/blob/9d42381d38fff23e9ef2c2d4de5a48efe937eb7a/lib/spack/spack/bootstrap.py#L213-L223

where we swap parts of the internal configuration to match what we want to enforce for bootstrapping. The issue stems from a combined action of having:
1. Configuration scopes swapped before the store
2. Store values being computed lazily

Since we swap the configuration scopes before needing to look at the store for the first time, when we swap the store we evaluate to the default values that have been just swapped in, rather than the ones that were originally in the user configuration.
This diff should be an hotfix to the issue:
```diff
diff --git a/lib/spack/spack/bootstrap.py b/lib/spack/spack/bootstrap.py
index 3d51ca931e..2cb4842465 100644
--- a/lib/spack/spack/bootstrap.py
+++ b/lib/spack/spack/bootstrap.py
@@ -216,9 +216,9 @@ def ensure_bootstrap_configuration():
         # Default configuration scopes excluding command line and builtin
         # but accounting for platform specific scopes
         config_scopes = _bootstrap_config_scopes()
-        with spack.config.use_configuration(*config_scopes):
-            with spack.repo.use_repositories(spack.paths.packages_path):
-                with spack.store.use_store(spack.paths.user_bootstrap_store):
+        with spack.repo.use_repositories(spack.paths.packages_path):
+            with spack.store.use_store(spack.paths.user_bootstrap_store):
+                with spack.config.use_configuration(*config_scopes):
                     with spack_python_interpreter():
                         yield

```

Resolving patch:
diff --git a/lib/spack/spack/bootstrap.py b/lib/spack/spack/bootstrap.py
--- a/lib/spack/spack/bootstrap.py
+++ b/lib/spack/spack/bootstrap.py
@@ -213,12 +213,12 @@ def _bootstrap_config_scopes():
 @contextlib.contextmanager
 def ensure_bootstrap_configuration():
     with spack.architecture.use_platform(spack.architecture.real_platform()):
-        # Default configuration scopes excluding command line and builtin
-        # but accounting for platform specific scopes
-        config_scopes = _bootstrap_config_scopes()
-        with spack.config.use_configuration(*config_scopes):
-            with spack.repo.use_repositories(spack.paths.packages_path):
-                with spack.store.use_store(spack.paths.user_bootstrap_store):
+        with spack.repo.use_repositories(spack.paths.packages_path):
+            with spack.store.use_store(spack.paths.user_bootstrap_store):
+                # Default configuration scopes excluding command line
+                # and builtin but accounting for platform specific scopes
+                config_scopes = _bootstrap_config_scopes()
+                with spack.config.use_configuration(*config_scopes):
                     with spack_python_interpreter():
                         yield
 
diff --git a/lib/spack/spack/store.py b/lib/spack/spack/store.py
--- a/lib/spack/spack/store.py
+++ b/lib/spack/spack/store.py
@@ -239,6 +239,19 @@ def _store_layout():
 layout = llnl.util.lang.LazyReference(_store_layout)
 
 
+def reinitialize():
+    """Restore globals to the same state they would have at start-up"""
+    global store
+    global root, unpadded_root, db, layout
+
+    store = llnl.util.lang.Singleton(_store)
+
+    root = llnl.util.lang.LazyReference(_store_root)
+    unpadded_root = llnl.util.lang.LazyReference(_store_unpadded_root)
+    db = llnl.util.lang.LazyReference(_store_db)
+    layout = llnl.util.lang.LazyReference(_store_layout)
+
+
 def retrieve_upstream_dbs():
     other_spack_instances = spack.config.get('upstreams', {})
 


Test patch:
diff --git a/lib/spack/spack/test/bootstrap.py b/lib/spack/spack/test/bootstrap.py
new file mode 100644
--- /dev/null
+++ b/lib/spack/spack/test/bootstrap.py
@@ -0,0 +1,26 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+import pytest
+
+import spack.bootstrap
+import spack.store
+
+
+@pytest.mark.regression('22294')
+def test_store_is_restored_correctly_after_bootstrap(mutable_config, tmpdir):
+    # Prepare a custom store path. This should be in a writeable location
+    # since Spack needs to initialize the DB.
+    user_path = str(tmpdir.join('store'))
+    # Reassign global variables in spack.store to the value
+    # they would have at Spack startup.
+    spack.store.reinitialize()
+    # Set the custom user path
+    spack.config.set('config:install_tree:root', user_path)
+
+    # Test that within the context manager we use the bootstrap store
+    # and that outside we restore the correct location
+    with spack.bootstrap.ensure_bootstrap_configuration():
+        assert spack.store.root == spack.paths.user_bootstrap_store
+    assert spack.store.root == user_path