Repository: spack/spack

Issue:
ASP solver: two unit tests failing on MacOS
Running the unit test suite reports two failures, both related to pickling objects, when testing with the new concretizer. The errors do not occur with the original concretizer.

### Steps to reproduce the issue
Setup `clingo` to be able to test the new concretizer. Then:
```console
$ SPACK_TEST_SOLVER=clingo spack unit-test lib/spack/spack/test/cmd/deprecate.py::test_deprecate_install lib/spack/spack/test/install.py::test_installed_upstream
```

### Error Message

The error message is the following:

<details>

<summary>SPACK_TEST_SOLVER=clingo spack unit-test ... </summary>

```
% SPACK_TEST_SOLVER=clingo spack unit-test lib/spack/spack/test/cmd/deprecate.py::test_deprecate_install lib/spack/spack/test/install.py::test_installed_upstream
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.8.5, pytest-3.2.5, py-1.4.34, pluggy-0.4.0
rootdir: /Users/culpo/PycharmProjects/spack, inifile: pytest.ini
collected 2 items                                                                                                                                                                                           

lib/spack/spack/test/cmd/deprecate.py F
lib/spack/spack/test/install.py F
========================================================================================= short test summary info ==========================================================================================
FAIL lib/spack/spack/test/cmd/deprecate.py::test_deprecate_install[mock_archive0]
FAIL lib/spack/spack/test/install.py::test_installed_upstream[mock_archive0]

======================================================================================== slowest 20 test durations =========================================================================================
1.78s call     lib/spack/spack/test/cmd/deprecate.py::test_deprecate_install[mock_archive0]
0.34s call     lib/spack/spack/test/install.py::test_installed_upstream[mock_archive0]
0.31s setup    lib/spack/spack/test/cmd/deprecate.py::test_deprecate_install[mock_archive0]
0.01s setup    lib/spack/spack/test/install.py::test_installed_upstream[mock_archive0]
0.01s teardown lib/spack/spack/test/cmd/deprecate.py::test_deprecate_install[mock_archive0]
0.00s teardown lib/spack/spack/test/install.py::test_installed_upstream[mock_archive0]
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________ test_deprecate_install[mock_archive0] ___________________________________________________________________________________

mock_packages = <spack.repo.RepoPath object at 0x102249220>
mock_archive = Archive(url='file:///private/var/folders/gy/g0z3m6dn0nz53nsttz2q0flm0000gn/T/pytest-of-culpo/pytest-84/mock-archive-di...sttz2q0flm0000gn/T/pytest-of-culpo/pytest-84/mock-archive-dir0/spack-src.tar.gz', expanded_archive_basedir='spack-src')
mock_fetch = None, install_mockery = None

    def test_deprecate_install(mock_packages, mock_archive, mock_fetch,
                               install_mockery):
        """Tests that the ```-i`` option allows us to deprecate in favor of a spec
        that is not yet installed."""
        install('libelf@0.8.10')
    
        to_deprecate = spack.store.db.query()
        assert len(to_deprecate) == 1
    
>       deprecate('-y', '-i', 'libelf@0.8.10', 'libelf@0.8.13')

lib/spack/spack/test/cmd/deprecate.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
lib/spack/spack/main.py:545: in __call__
    self.returncode = _invoke_command(
lib/spack/spack/main.py:490: in _invoke_command
    return_val = command(parser, args)
lib/spack/spack/cmd/deprecate.py:129: in deprecate
    dcate.package.do_deprecate(dcator, link_fn)
lib/spack/spack/package.py:2229: in do_deprecate
    deprecator.package.do_install()
lib/spack/spack/package.py:1697: in do_install
    builder.install()
lib/spack/spack/installer.py:1534: in install
    self._install_task(task)
lib/spack/spack/installer.py:1122: in _install_task
    spack.build_environment.start_build_process(
lib/spack/spack/build_environment.py:945: in start_build_process
    serialized_pkg = spack.subprocess_context.PackageInstallContext(pkg)
lib/spack/spack/subprocess_context.py:69: in __init__
    self.serialized_pkg = serialize(pkg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <spack.pkg.builtin.mock.libelf.Libelf object at 0x102c253d0>

    def serialize(obj):
        serialized_obj = io.BytesIO()
>       pickle.dump(obj, serialized_obj)
E       _pickle.PicklingError: Can't pickle <class 'llnl.util.lang.Spec'>: attribute lookup Spec on llnl.util.lang failed

lib/spack/spack/subprocess_context.py:43: PicklingError
------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------
==> Fetching file:///private/var/folders/gy/g0z3m6dn0nz53nsttz2q0flm0000gn/T/pytest-of-culpo/pytest-84/mock-archive-dir0/spack-src.tar.gz
==> libelf: Executing phase: 'install'
==> libelf: Successfully installed libelf-0.8.10-gstytvv2sv35kdf6yw2uhg4wcankkprh
  Fetch: 0.02s.  Build: 0.47s.  Total: 0.49s.
[+] /private/var/folders/gy/g0z3m6dn0nz53nsttz2q0flm0000gn/T/pytest-of-culpo/pytest-84/test_deprecate_install_mock_ar0/opt/test-debian6-core2/gcc-4.5.0/libelf-0.8.10-gstytvv2sv35kdf6yw2uhg4wcankkprh
------------------------------------------------------------------------------------------- Captured stderr call -------------------------------------------------------------------------------------------
/Users/culpo/PycharmProjects/spack/lib/spack/spack/solver/concretize.lp:153:42-77: info: atom does not occur in any rule head:
  possible_provider(Package,Virtual)

/Users/culpo/PycharmProjects/spack/lib/spack/spack/solver/concretize.lp:153:42-77: info: atom does not occur in any rule head:
  possible_provider(Package,Virtual)

__________________________________________________________________________________ test_installed_upstream[mock_archive0] __________________________________________________________________________________

tmpdir_factory = <_pytest.tmpdir.TempdirFactory object at 0x101fefb20>, install_mockery = None, mock_fetch = None, gen_mock_layout = <function gen_mock_layout.<locals>.create_layout at 0x1021b73a0>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x102970730>

    def test_installed_upstream(tmpdir_factory, install_mockery, mock_fetch,
                                gen_mock_layout, monkeypatch):
        """Check that when a dependency package is recorded as installed in
           an upstream database that it is not reinstalled.
        """
        mock_db_root = str(tmpdir_factory.mktemp('mock_db_root'))
        prepared_db = spack.database.Database(mock_db_root)
    
        upstream_layout = gen_mock_layout('/a/')
    
        dependency = spack.spec.Spec('dependency-install')
        dependency.concretize()
        prepared_db.add(dependency, upstream_layout)
    
        downstream_db_root = str(
            tmpdir_factory.mktemp('mock_downstream_db_root'))
        db_for_test = spack.database.Database(
            downstream_db_root, upstream_dbs=[prepared_db])
        monkeypatch.setattr(spack.store, 'db', db_for_test)
        dependent = spack.spec.Spec('dependent-install')
        dependent.concretize()
    
        new_dependency = dependent['dependency-install']
        assert new_dependency.package.installed_upstream
        assert (new_dependency.prefix ==
                upstream_layout.path_for_spec(dependency))
    
>       dependent.package.do_install()

lib/spack/spack/test/install.py:246: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
lib/spack/spack/package.py:1697: in do_install
    builder.install()
lib/spack/spack/installer.py:1534: in install
    self._install_task(task)
lib/spack/spack/installer.py:1122: in _install_task
    spack.build_environment.start_build_process(
lib/spack/spack/build_environment.py:945: in start_build_process
    serialized_pkg = spack.subprocess_context.PackageInstallContext(pkg)
lib/spack/spack/subprocess_context.py:69: in __init__
    self.serialized_pkg = serialize(pkg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <spack.pkg.builtin.mock.dependent-install.DependentInstall object at 0x1020ce1f0>

    def serialize(obj):
        serialized_obj = io.BytesIO()
>       pickle.dump(obj, serialized_obj)
E       _pickle.PicklingError: Can't pickle <class 'llnl.util.lang.Spec'>: attribute lookup Spec on llnl.util.lang failed

lib/spack/spack/subprocess_context.py:43: PicklingError
------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------
[+] /private/var/folders/gy/g0z3m6dn0nz53nsttz2q0flm0000gn/T/pytest-of-culpo/pytest-84/test_installed_upstream_mock_a0/a/dependency-install
==> Installing dependent-install-2.0-erjlafcwgpyioat5tfiedpy6nu3ohp5h
==> No binary for dependent-install-2.0-erjlafcwgpyioat5tfiedpy6nu3ohp5h found: installing from source
------------------------------------------------------------------------------------------- Captured stderr call -------------------------------------------------------------------------------------------
/Users/culpo/PycharmProjects/spack/lib/spack/spack/solver/concretize.lp:153:42-77: info: atom does not occur in any rule head:
  possible_provider(Package,Virtual)

/Users/culpo/PycharmProjects/spack/lib/spack/spack/solver/concretize.lp:153:42-77: info: atom does not occur in any rule head:
  possible_provider(Package,Virtual)

==> Error: Failed to install dependent-install due to PicklingError: Can't pickle <class 'llnl.util.lang.Spec'>: attribute lookup Spec on llnl.util.lang failed
========================================================================================= 2 failed in 2.84 seconds =========================================================================================
```

</details>

### Information on your system

* **Spack:** 0.16.0-918-ec7caefc69
* **Python:** 3.8.5
* **Platform:** darwin-bigsur-cannonlake
* **Concretizer:** clingo

The exact version of the OS is 11.1 (20C69)

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output



Discussion:
These errors have been unnoticed since we don't test the new ASP based solver with MacOS, mainly due to the long setup time it would take for each run to install clingo (for linux we use a container with clingo pre-installed). We should probably add a CI run to cover this gap while are working on bootstrapping `clingo`. 
> 
> 
> These errors have been unnoticed since we don't test the new ASP based solver with MacOS, mainly due to the long setup time it would take for each run to install clingo (for linux we use a container with clingo pre-installed). We should probably add a CI run to cover this gap while are working on bootstrapping `clingo`.

For what it's worth, merging #20657 would allow for the use of clingo binary wheels via `pip` which would speed up the process on OSX.
Bumped this from impact-low to impact-medium since clingo will soon be the default concretizer and macOS support is important (at least to me 😆 )
Possibly a duplicate of #23892. Not saying we should close either issue, but if we fix one we might fix the other.
Added a temporary workaround in #23892 if anyone else encounters this issue before we get it fixed.
I'm not sure this is a duplicate. Here `pickle` is not able to find `Spec` in `llnl.util.lang` 🤔 Still not sure why it searches that there.
Note to self, it's very likely the:

https://github.com/spack/spack/blob/4ddc0ff218d5ed0968f6efb0fac855b1b7127375/lib/spack/llnl/util/lang.py#L752-L759

that is confusing `pickle` sometimes. 

Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1019,7 +1019,9 @@ class SpecBuildInterface(lang.ObjectWrapper):
 
     def __init__(self, spec, name, query_parameters):
         super(SpecBuildInterface, self).__init__(spec)
-
+        # Adding new attributes goes after super() call since the ObjectWrapper
+        # resets __dict__ to behave like the passed object
+        self.token = spec, name, query_parameters
         is_virtual = spack.repo.path.is_virtual(name)
         self.last_query = QueryState(
             name=name,
@@ -1027,6 +1029,9 @@ def __init__(self, spec, name, query_parameters):
             isvirtual=is_virtual
         )
 
+    def __reduce__(self):
+        return SpecBuildInterface, self.token
+
 
 @lang.lazy_lexicographic_ordering(set_hash=False)
 class Spec(object):


Test patch:
diff --git a/.github/workflows/unit_tests.yaml b/.github/workflows/unit_tests.yaml
--- a/.github/workflows/unit_tests.yaml
+++ b/.github/workflows/unit_tests.yaml
@@ -337,10 +337,14 @@ jobs:
       run: |
         brew install dash fish gcc gnupg2 kcov
     - name: Run unit tests
+      env:
+        SPACK_TEST_SOLVER: clingo
       run: |
         git --version
         . .github/workflows/setup_git.sh
         . share/spack/setup-env.sh
+        $(which spack) bootstrap untrust spack-install
+        $(which spack) solve zlib
         if [ "${{ needs.changes.outputs.with_coverage }}" == "true" ]
         then
           coverage run $(which spack) unit-test -x