Repository: spack/spack

Issue:
spack test doesn't support Python 3.8 + macOS
The `spack test` command doesn't seem to work with Python 3, at least for me:
```console
$ python3 $(which spack) test run m4
==> Spack test 5zpwbxxu43yidobk5y7qwagzxo77jeyx
$ python3 $(which spack) test results --logs
==> Results for test suite '5zpwbxxu43yidobk5y7qwagzxo77jeyx':
==>   m4-1.4.18-kzezvfi FAILED

==>   m4-1.4.18-qlyhwnr FAILED

$ python2 $(which spack) test run m4
==> Spack test 5zpwbxxu43yidobk5y7qwagzxo77jeyx
==> Testing package m4-1.4.18-kzezvfi
==> Testing package m4-1.4.18-qlyhwnr
$ python2 $(which spack) test results --logs
==> Results for test suite '5zpwbxxu43yidobk5y7qwagzxo77jeyx':
==>   m4-1.4.18-kzezvfi PASSED
==> Testing package m4-1.4.18-kzezvfi
==> [2020-11-20-16:11:31.081381] test: ensuring m4 version is 1.4.18
==> [2020-11-20-16:11:31.085485] '/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/m4-1.4.18-kzezvfiholjl32zfa2vfkmvycrcp7rq6/bin/m4' '--version'
m4 (GNU M4) 1.4.18
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Rene' Seindal.
PASSED
==> [2020-11-20-16:11:31.155946] test: ensuring m4 example succeeds
==> [2020-11-20-16:11:31.157026] '/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/apple-clang-12.0.0/m4-1.4.18-kzezvfiholjl32zfa2vfkmvycrcp7rq6/bin/m4' '/Users/Adam/.spack/test/5zpwbxxu43yidobk5y7qwagzxo77jeyx/m4-1.4.18-kzezvfi/data/m4/hello.m4'

// macro is defined
Hello, World!
PASSED

==>   m4-1.4.18-qlyhwnr PASSED
==> Testing package m4-1.4.18-qlyhwnr
==> [2020-11-20-16:11:47.213468] test: ensuring m4 version is 1.4.18
==> [2020-11-20-16:11:47.220733] '/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/gcc-10.2.0/m4-1.4.18-qlyhwnrpg6e66gdldgivjps47fyu6gbn/bin/m4' '--version'
m4 (GNU M4) 1.4.18
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Rene' Seindal.
PASSED
==> [2020-11-20-16:11:47.316219] test: ensuring m4 example succeeds
==> [2020-11-20-16:11:47.317434] '/Users/Adam/spack/opt/spack/darwin-catalina-x86_64/gcc-10.2.0/m4-1.4.18-qlyhwnrpg6e66gdldgivjps47fyu6gbn/bin/m4' '/Users/Adam/.spack/test/5zpwbxxu43yidobk5y7qwagzxo77jeyx/m4-1.4.18-qlyhwnr/data/m4/hello.m4'

// macro is defined
Hello, World!
PASSED
```

### Information on your system

$ spack debug report
* **Spack:** 0.16.0-32-3fbda07a87
* **Python:** 3.8.6
* **Platform:** darwin-catalina-ivybridge
* **Concretizer:** original

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output
RecursionError: maximum recursion depth exceeded while pickling an object
I encountered a build error on macOS that only seems to happen with Python 3. I'm guessing this is one of those Python 3.7+ macOS multiprocessing bugs we've encountered in the past.

### Steps to reproduce the issue

I'm trying to install the same environment as #23891.

### Error Message

```
==> [2021-05-25-12:10:42.975111, 54459] 54459: Installing py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw
==> [2021-05-25-12:10:42.975262, 54459] Searching for binary cache of py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw
==> [2021-05-25-12:10:42.975652, 54459] Reading config file /Users/Adam/spack/etc/spack/defaults/mirrors.yaml
==> [2021-05-25-12:10:43.314505, 54459] Did not find darwin-catalina-x86_64-apple-clang-12.0.0-py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw.spec.yaml on https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache/darwin-catalina-x86_64-apple-clang-12.0.0-py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw.spec.yaml
  Download failed: HTTP Error 404: Not Found
==> [2021-05-25-12:10:43.319141, 54459] No binary for py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw found: installing from source
==> [2021-05-25-12:10:43.321732, 54459] Reading config file /Users/Adam/spack/etc/spack/defaults/packages.yaml
==> [2021-05-25-12:10:43.367691, 54459] Reading config file /Users/Adam/spack/etc/spack/defaults/darwin/packages.yaml
==> [2021-05-25-12:10:43.383541, 54459] Reading config file /Users/Adam/.spack/packages.yaml
==> [2021-05-25-12:10:43.572503, 54459] Warning: Missing a source id for apple-libuuid@1353.100.2
==> [2021-05-25-12:10:43.895044, 54459] '/usr/bin/git' 'describe' '--tags' '--match' 'v*'
==> [2021-05-25-12:10:44.001211, 54459] Flagging py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw as failed: maximum recursion depth exceeded while pickling an object
==> [2021-05-25-12:10:44.012687, 54459] Warning: Skipping build of py-torchvision-0.9.1-2w3ky2qzjz5lverygxah2raqclk2jqvz since py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw failed
==> [2021-05-25-12:10:44.012788, 54459] Flagging py-torchvision-0.9.1-2w3ky2qzjz5lverygxah2raqclk2jqvz as failed
==> [2021-05-25-12:10:44.023099, 54459] Error: Failed to install py-torch due to RecursionError: maximum recursion depth exceeded while pickling an object
```

### Information on your system

* **Spack:** 0.16.1-2888-a284edb852
* **Python:** 3.8.10
* **Platform:** darwin-catalina-ivybridge
* **Concretizer:** clingo

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output


Discussion:
@scheibelp I've narrowed this down a bit. With Python 2.7 and 3.7, I'm unable to reproduce this. But with Python 3.8 and macOS, I can. This looks like another Python 3.8 + macOS bug in develop.
Ping @scheibelp @becker33, can we bump this up in priority?
Has anyone had a chance to work on this? It keeps getting bumped from one release to the next.
@adamjstewart I'm not having the problem with the latest `develop` on my Mac (Big Sur, Python 3.9.5 and 2.7.16).  I see this is listed in the `0.16.3` and you were using `0.16.0`.  Should I be testing with a specific release or commit?
Oh interesting, maybe it was fixed recently. I'll investigate. 
I'm so confused, I can't even reproduce this on the 0.16.0 release anymore. I guess we can close this for now until someone encounters it again and figures out how to reproduce.
Just reproduced this again. Going to try to figure out what changed. I'm now in an environment with a Spack-built Python, so that may affect things.
Okay, if I remove the try-except logic in `spack.install_test.TestSuite.__call__`, I get the same error message as in #23892:
```
==> Error: maximum recursion depth exceeded while pickling an object
```
@alalazo just fixed this for installations in #25658, looks like we need to add this to another method to fix it for smoke tests.
Tried adding a `TestSuite.__reduce__` method but it didn't solve the issue, so the issue isn't with pickling `TestSuite` but some other object.
Figured out how to figure out which object is having trouble being pickled. After removing the try-except logic:
```console
$ spack --stacktrace test run
...
Traceback (most recent call last):
  File "/Users/Adam/spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/Users/Adam/spack/lib/spack/spack/main.py", line 797, in main
    return _invoke_command(command, parser, args, unknown)
  File "/Users/Adam/spack/lib/spack/spack/main.py", line 525, in _invoke_command
    return_val = command(parser, args)
  File "/Users/Adam/spack/lib/spack/spack/cmd/test.py", line 411, in test
    globals()['test_%s' % args.test_command](args)
  File "/Users/Adam/spack/lib/spack/spack/cmd/test.py", line 193, in test_run
    test_suite(remove_directory=not args.keep_stage,
  File "/Users/Adam/spack/lib/spack/spack/install_test.py", line 129, in __call__
    spec.package.do_test(
  File "/Users/Adam/spack/lib/spack/spack/package.py", line 1791, in do_test
    spack.build_environment.start_build_process(self, test_process, kwargs)
  File "/Users/Adam/spack/lib/spack/spack/build_environment.py", line 1111, in start_build_process
    serialized_pkg = spack.subprocess_context.PackageInstallContext(pkg)
  File "/Users/Adam/spack/lib/spack/spack/subprocess_context.py", line 69, in __init__
    self.serialized_env = serialize(ev._active_environment)
  File "/Users/Adam/spack/lib/spack/spack/subprocess_context.py", line 41, in serialize
    pickle.dump(obj, serialized_obj)
RecursionError: maximum recursion depth exceeded while pickling an object
```
If I print the object we're trying to serialize, it gives me `spack.environment.Environment`. Going to try to add a `__reduce__` method to that.
Hit this again today
Current workaround is to use:
```python
import sys

sys.setrecursionlimit(10000)
```
somewhere in Spack, but this isn't a great long-term solution unless we know why the issue is occuring.
After concretizing the environment, this can be reproduced in an intepreter like:
```python
>>> import io
>>> import pickle
>>> import spack.environment
>>> with spack.environment.Environment('./') as env:
...     s = next(iter(env.all_specs()))
...     stream = io.BytesIO()
...     pickle.dump(s.package, stream)
```
Ugh, [this](https://stackoverflow.com/questions/2134706/hitting-maximum-recursion-depth-using-pickle-cpickle) is not promising. Maybe though we can pickle things differently than just using defaults.

Resolving patch:
diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py
--- a/lib/spack/spack/environment.py
+++ b/lib/spack/spack/environment.py
@@ -581,6 +581,10 @@ def regenerate(self, all_specs, roots):
                     tty.warn(msg)
 
 
+def _create_environment(*args, **kwargs):
+    return Environment(*args, **kwargs)
+
+
 class Environment(object):
     def __init__(self, path, init_file=None, with_view=None, keep_relative=False):
         """Create a new environment.
@@ -601,6 +605,9 @@ def __init__(self, path, init_file=None, with_view=None, keep_relative=False):
                 directory.
         """
         self.path = os.path.abspath(path)
+        self.init_file = init_file
+        self.with_view = with_view
+        self.keep_relative = keep_relative
 
         self.txlock = lk.Lock(self._transaction_lock_path)
 
@@ -643,6 +650,11 @@ def __init__(self, path, init_file=None, with_view=None, keep_relative=False):
         # If with_view is None, then defer to the view settings determined by
         # the manifest file
 
+    def __reduce__(self):
+        return _create_environment, (
+            self.path, self.init_file, self.with_view, self.keep_relative
+        )
+
     def _rewrite_relative_paths_on_relocation(self, init_file_dir):
         """When initializing the environment from a manifest file and we plan
            to store the environment in a different directory, we have to rewrite


Test patch:
diff --git a/lib/spack/spack/test/environment.py b/lib/spack/spack/test/environment.py
new file mode 100644
--- /dev/null
+++ b/lib/spack/spack/test/environment.py
@@ -0,0 +1,15 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import pickle
+
+from spack.environment import Environment
+
+
+def test_environment_pickle(tmpdir):
+    env1 = Environment(str(tmpdir))
+    obj = pickle.dumps(env1)
+    env2 = pickle.loads(obj)
+    assert isinstance(env2, Environment)