Repository: spack/spack

Issue:
RecursionError: maximum recursion depth exceeded while pickling an object
I encountered a build error on macOS that only seems to happen with Python 3. I'm guessing this is one of those Python 3.7+ macOS multiprocessing bugs we've encountered in the past.

### Steps to reproduce the issue

I'm trying to install the same environment as #23891.

### Error Message

```
==> [2021-05-25-12:10:42.975111, 54459] 54459: Installing py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw
==> [2021-05-25-12:10:42.975262, 54459] Searching for binary cache of py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw
==> [2021-05-25-12:10:42.975652, 54459] Reading config file /Users/Adam/spack/etc/spack/defaults/mirrors.yaml
==> [2021-05-25-12:10:43.314505, 54459] Did not find darwin-catalina-x86_64-apple-clang-12.0.0-py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw.spec.yaml on https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache/darwin-catalina-x86_64-apple-clang-12.0.0-py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw.spec.yaml
  Download failed: HTTP Error 404: Not Found
==> [2021-05-25-12:10:43.319141, 54459] No binary for py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw found: installing from source
==> [2021-05-25-12:10:43.321732, 54459] Reading config file /Users/Adam/spack/etc/spack/defaults/packages.yaml
==> [2021-05-25-12:10:43.367691, 54459] Reading config file /Users/Adam/spack/etc/spack/defaults/darwin/packages.yaml
==> [2021-05-25-12:10:43.383541, 54459] Reading config file /Users/Adam/.spack/packages.yaml
==> [2021-05-25-12:10:43.572503, 54459] Warning: Missing a source id for apple-libuuid@1353.100.2
==> [2021-05-25-12:10:43.895044, 54459] '/usr/bin/git' 'describe' '--tags' '--match' 'v*'
==> [2021-05-25-12:10:44.001211, 54459] Flagging py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw as failed: maximum recursion depth exceeded while pickling an object
==> [2021-05-25-12:10:44.012687, 54459] Warning: Skipping build of py-torchvision-0.9.1-2w3ky2qzjz5lverygxah2raqclk2jqvz since py-torch-1.8.1-lb6hngapkdw4k6jvb6skwtaxeay2exfw failed
==> [2021-05-25-12:10:44.012788, 54459] Flagging py-torchvision-0.9.1-2w3ky2qzjz5lverygxah2raqclk2jqvz as failed
==> [2021-05-25-12:10:44.023099, 54459] Error: Failed to install py-torch due to RecursionError: maximum recursion depth exceeded while pickling an object
```

### Information on your system

* **Spack:** 0.16.1-2888-a284edb852
* **Python:** 3.8.10
* **Platform:** darwin-catalina-ivybridge
* **Concretizer:** clingo

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output


Discussion:
Hit this again today
Current workaround is to use:
```python
import sys

sys.setrecursionlimit(10000)
```
somewhere in Spack, but this isn't a great long-term solution unless we know why the issue is occuring.
After concretizing the environment, this can be reproduced in an intepreter like:
```python
>>> import io
>>> import pickle
>>> import spack.environment
>>> with spack.environment.Environment('./') as env:
...     s = next(iter(env.all_specs()))
...     stream = io.BytesIO()
...     pickle.dump(s.package, stream)
```
Ugh, [this](https://stackoverflow.com/questions/2134706/hitting-maximum-recursion-depth-using-pickle-cpickle) is not promising. Maybe though we can pickle things differently than just using defaults.

Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -2093,28 +2093,7 @@ def from_dict(data):
         Parameters:
         data -- a nested dict/list data structure read from YAML or JSON.
         """
-        nodes = data['spec']
-
-        # Read nodes out of list.  Root spec is the first element;
-        # dependencies are the following elements.
-        dep_list = [Spec.from_node_dict(node) for node in nodes]
-        if not dep_list:
-            raise spack.error.SpecError("YAML spec contains no nodes.")
-        deps = dict((spec.name, spec) for spec in dep_list)
-        spec = dep_list[0]
-
-        for node in nodes:
-            # get dependency dict from the node.
-            name = next(iter(node))
-
-            if 'dependencies' not in node[name]:
-                continue
-
-            yaml_deps = node[name]['dependencies']
-            for dname, dhash, dtypes in Spec.read_yaml_dep_specs(yaml_deps):
-                deps[name]._add_dependency(deps[dname], dtypes)
-
-        return spec
+        return _spec_from_dict(data)
 
     @staticmethod
     def from_yaml(stream):
@@ -4385,6 +4364,43 @@ def __hash__(self):
         # so we hope it only runs on abstract specs, which are small.
         return hash(lang.tuplify(self._cmp_iter))
 
+    def __reduce__(self):
+        return _spec_from_dict, (self.to_dict(hash=ht.build_hash),)
+
+
+# Note: This function has been refactored from being a static method
+# of Spec to be a function at the module level. This was needed to
+# support its use in __reduce__ to pickle a Spec object in Python 2.
+# It can be moved back safely after we drop support for Python 2.7
+def _spec_from_dict(data):
+    """Construct a spec from YAML.
+
+    Parameters:
+    data -- a nested dict/list data structure read from YAML or JSON.
+    """
+    nodes = data['spec']
+
+    # Read nodes out of list.  Root spec is the first element;
+    # dependencies are the following elements.
+    dep_list = [Spec.from_node_dict(node) for node in nodes]
+    if not dep_list:
+        raise spack.error.SpecError("YAML spec contains no nodes.")
+    deps = dict((spec.name, spec) for spec in dep_list)
+    spec = dep_list[0]
+
+    for node in nodes:
+        # get dependency dict from the node.
+        name = next(iter(node))
+
+        if 'dependencies' not in node[name]:
+            continue
+
+        yaml_deps = node[name]['dependencies']
+        for dname, dhash, dtypes in Spec.read_yaml_dep_specs(yaml_deps):
+            deps[name]._add_dependency(deps[dname], dtypes)
+
+    return spec
+
 
 class LazySpecCache(collections.defaultdict):
     """Cache for Specs that uses a spec_like as key, and computes lazily


Test patch: