Repository: pypa/setuptools

Issue:
package metadata is no longer picklable
Since switching to OrderedDict, package metadata can no longer be pickled.

This can cause problems if you try to use setuptools with multiprocessing.

```
Traceback (most recent call last):
  File "/Users/dan/Documents/colcon_ws/src/colcon-python-setup-py/colcon_python_setup_py/package_identification/python_setup_py.py", line 257, in get_setup_information
    'stop_after': 'config'
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 261, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<distutils.dist.DistributionMetadata object at 0x10c1fcd50>'. Reason: 'PicklingError("Can't pickle <class 'setuptools._vendor.ordered_set.OrderedSet'>: it's not the same object as setuptools._vendor.ordered_set.OrderedSet")'
```

_Originally posted by @rotu in https://github.com/pypa/setuptools/pull/1690#issuecomment-545992670_
package metadata is no longer picklable
Since switching to OrderedDict, package metadata can no longer be pickled.

This can cause problems if you try to use setuptools with multiprocessing.

```
Traceback (most recent call last):
  File "/Users/dan/Documents/colcon_ws/src/colcon-python-setup-py/colcon_python_setup_py/package_identification/python_setup_py.py", line 257, in get_setup_information
    'stop_after': 'config'
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 261, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<distutils.dist.DistributionMetadata object at 0x10c1fcd50>'. Reason: 'PicklingError("Can't pickle <class 'setuptools._vendor.ordered_set.OrderedSet'>: it's not the same object as setuptools._vendor.ordered_set.OrderedSet")'
```

_Originally posted by @rotu in https://github.com/pypa/setuptools/pull/1690#issuecomment-545992670_


Discussion:
Minimal example:

```
import multiprocessing

def f():
	import setuptools
	return setuptools.Distribution().metadata.provides_extras

with multiprocessing.Pool(1) as p:
	p.apply(f)
```
I think this is due to vendor module trickery here: https://github.com/pypa/setuptools/blob/297f2adceda3af402fc08311e42505c8cdc9c54b/setuptools/extern/__init__.py
I'm on Ubuntu 18.04

Possibly useful exploratory code:
```
import setuptools
from setuptools.extern.ordered_set import OrderedSet as os_extern
from setuptools._vendor.ordered_set import OrderedSet as os_vendor

e = setuptools.Distribution().metadata.provides_extras
print(os_extern is os_vendor)  # False
print(type(e).__module__)  # setuptools._vendor.ordered_set
print(type(e) is os_extern)  # True
print(type(e) is os_vendor)  # False
```
Minimal example:

```
import multiprocessing

def f():
	import setuptools
	return setuptools.Distribution().metadata.provides_extras

with multiprocessing.Pool(1) as p:
	p.apply(f)
```
I think this is due to vendor module trickery here: https://github.com/pypa/setuptools/blob/297f2adceda3af402fc08311e42505c8cdc9c54b/setuptools/extern/__init__.py
I'm on Ubuntu 18.04

Possibly useful exploratory code:
```
import setuptools
from setuptools.extern.ordered_set import OrderedSet as os_extern
from setuptools._vendor.ordered_set import OrderedSet as os_vendor

e = setuptools.Distribution().metadata.provides_extras
print(os_extern is os_vendor)  # False
print(type(e).__module__)  # setuptools._vendor.ordered_set
print(type(e) is os_extern)  # True
print(type(e) is os_vendor)  # False
```

Resolving patch:
diff --git a/pkg_resources/extern/__init__.py b/pkg_resources/extern/__init__.py
--- a/pkg_resources/extern/__init__.py
+++ b/pkg_resources/extern/__init__.py
@@ -43,13 +43,6 @@ def load_module(self, fullname):
                 __import__(extant)
                 mod = sys.modules[extant]
                 sys.modules[fullname] = mod
-                # mysterious hack:
-                # Remove the reference to the extant package/module
-                # on later Python versions to cause relative imports
-                # in the vendor package to resolve the same modules
-                # as those going through this importer.
-                if prefix and sys.version_info > (3, 3):
-                    del sys.modules[extant]
                 return mod
             except ImportError:
                 pass
diff --git a/setuptools/extern/__init__.py b/setuptools/extern/__init__.py
--- a/setuptools/extern/__init__.py
+++ b/setuptools/extern/__init__.py
@@ -43,13 +43,6 @@ def load_module(self, fullname):
                 __import__(extant)
                 mod = sys.modules[extant]
                 sys.modules[fullname] = mod
-                # mysterious hack:
-                # Remove the reference to the extant package/module
-                # on later Python versions to cause relative imports
-                # in the vendor package to resolve the same modules
-                # as those going through this importer.
-                if sys.version_info >= (3, ):
-                    del sys.modules[extant]
                 return mod
             except ImportError:
                 pass


Test patch:
diff --git a/setuptools/tests/test_extern.py b/setuptools/tests/test_extern.py
new file mode 100644
--- /dev/null
+++ b/setuptools/tests/test_extern.py
@@ -0,0 +1,22 @@
+import importlib
+import pickle
+
+from setuptools import Distribution
+from setuptools.extern import ordered_set
+from setuptools.tests import py3_only
+
+
+def test_reimport_extern():
+    ordered_set2 = importlib.import_module(ordered_set.__name__)
+    assert ordered_set is ordered_set2
+
+
+def test_orderedset_pickle_roundtrip():
+    o1 = ordered_set.OrderedSet([1, 2, 5])
+    o2 = pickle.loads(pickle.dumps(o1))
+    assert o1 == o2
+
+
+@py3_only
+def test_distribution_picklable():
+    pickle.loads(pickle.dumps(Distribution()))