Repository: pypa/setuptools

Issue:
Plugins using setuptools entry points resolve to system and not setup_requires
Assume that you have e.g. setuptools_scm 3 installed system-wide, and try to to run the following setup.py:
```python
from setuptools import setup
setup(name="foo", setup_requires=["setuptools_scm>=4"])
```
you get the warning
```
/usr/lib/python3.8/site-packages/setuptools/dist.py:701: UserWarning: Module setuptools_scm was already imported from /usr/lib/python3.8/site-packages/setuptools_scm/__init__.py, but /tmp/foo/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
```
(i.e. the system-wide setuptools_scm was loaded, even though it is too old).  One can confirm that the "correct" newer setuptools_scm indeed doesn't get used by changing setup.py to
```python
from setuptools import setup
setup(name="foo",
      setup_requires=["setuptools_scm>=4"],
      use_scm_version={"parentdir_prefix_version": "foo-"})
```
in which case one gets the same warning and then a traceback
```
TypeError: __init__() got an unexpected keyword argument 'parentdir_prefix_version'
```

I guess a solution may be e.g. to move the entry point (which is what gets eagerly loaded by setuptools) to a separate package (let's say `_setuptools_scm_helper`) which itself imports setuptools_scm only when actually needed with a pass-through shim so that setuptools only imports `_setuptools_scm_helper` and then the correct version of setuptools_scm can be loaded when needed.
Plugins using setuptools entry points resolve to system and not setup_requires
Assume that you have e.g. setuptools_scm 3 installed system-wide, and try to to run the following setup.py:
```python
from setuptools import setup
setup(name="foo", setup_requires=["setuptools_scm>=4"])
```
you get the warning
```
/usr/lib/python3.8/site-packages/setuptools/dist.py:701: UserWarning: Module setuptools_scm was already imported from /usr/lib/python3.8/site-packages/setuptools_scm/__init__.py, but /tmp/foo/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
```
(i.e. the system-wide setuptools_scm was loaded, even though it is too old).  One can confirm that the "correct" newer setuptools_scm indeed doesn't get used by changing setup.py to
```python
from setuptools import setup
setup(name="foo",
      setup_requires=["setuptools_scm>=4"],
      use_scm_version={"parentdir_prefix_version": "foo-"})
```
in which case one gets the same warning and then a traceback
```
TypeError: __init__() got an unexpected keyword argument 'parentdir_prefix_version'
```

I guess a solution may be e.g. to move the entry point (which is what gets eagerly loaded by setuptools) to a separate package (let's say `_setuptools_scm_helper`) which itself imports setuptools_scm only when actually needed with a pass-through shim so that setuptools only imports `_setuptools_scm_helper` and then the correct version of setuptools_scm can be loaded when needed.


Discussion:
@jaraco does this relate to the keyword hook usage? 
@anntzer whats the setuptools version?

usually this is something i would take a look at on a weekend, but the next 2 weekends i am preoccupied with personal matters
This is with setuptools 46.4.0.
No hurries :)
Setuptools_scm and other setuptools plugins are not intended to be installed system-wide. Why not just not do that?
> does this relate to the keyword hook usage?

I'm not sure, but you can find out by testing with `setuptools<42` and see if the behavior is any different. It's possible that it is affected, because the `finalize_distribution_options` hook does run (and thus enumerate entry points) earlier than other hooks.
I had not ever seen that recommendation anywhere (I would be happy to see guidance to that effect).
I'm happy to take your word for it, but then 1) perhaps setuptools/pip could somehow recognize that (e.g. packages providing setuptools-related entry points) and indeed refuse to install them, and 2) should e.g. linux/conda packagers get a warning requesting them to remove the packages (https://packages.debian.org/sid/python3-setuptools-scm, https://www.archlinux.org/packages/community/any/python-setuptools-scm/, https://anaconda.org/conda-forge/setuptools_scm etc.)?
Now that I think about it, your best option is probably to use pip's build isolation to bypass system packages altogether. `pip --use-pep517 foo` should work.
PEP517 is not really an option for us because we need to support editable installs (because the package takes a while to compile).
By "we" I specifically meant Matplotlib, in this case; I was hoping to switch it from versioneer to setuptools_scm.
I'm able to replicate the issue with:

```
draft $ cat > setup.py
__import__('setuptools').setup(name='foo', setup_requires=['setuptools_scm>=4'])
draft $ pip-run -q 'setuptools_scm<4' -- setup.py --name
WARNING: The wheel package is not available.
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py:701: UserWarning: Module setuptools_scm was already imported from /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-0cfuhayx/setuptools_scm/__init__.py, but /Users/jaraco/draft/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
foo
```
Confirmed - the issue emerges at setuptools 42.

```
draft $ pip-run -q 'setuptools_scm<4' 'setuptools<42' -- setup.py --name
foo
draft $ pip-run -q 'setuptools_scm<4' 'setuptools==42' -- setup.py --name
WARNING: The wheel package is not available.
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-f_0y2mmd/setuptools/dist.py:724: UserWarning: Module setuptools_scm was already imported from /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-f_0y2mmd/setuptools_scm/__init__.py, but /Users/jaraco/draft/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
foo
```
I've figured out the issue. When `setup()` is called, it fires up a minimal `Distribution()` class, just enough to fetch the build requirements. It then adds those to the working set and proceeds to run `setup()` normally.

However, because constructing a `Distribution` triggers `finalize_distribution_options`, that causes plugins to get initialized and thus things like `setuptools_scm` to load before `setup_requires` was honored.

This patch works around the issue:

```
diff --git a/setuptools/__init__.py b/setuptools/__init__.py
index 811f3fd2..dbcb1723 100644
--- a/setuptools/__init__.py
+++ b/setuptools/__init__.py
@@ -128,7 +128,11 @@ if PY3:
 def _install_setup_requires(attrs):
     # Note: do not use `setuptools.Distribution` directly, as
     # our PEP 517 backend patch `distutils.core.Distribution`.
-    dist = distutils.core.Distribution(dict(
+    class NeuteredDistribution(distutils.core.Distribution):
+        def finalize_options(self):
+            pass
+
+    dist = NeuteredDistribution(dict(
         (k, v) for k, v in attrs.items()
         if k in ('dependency_links', 'setup_requires')
     ))
```
I've transferred the issue to the setuptools project, as setuptools_scm is just one plugin affected.
@jaraco does this relate to the keyword hook usage? 
@anntzer whats the setuptools version?

usually this is something i would take a look at on a weekend, but the next 2 weekends i am preoccupied with personal matters
This is with setuptools 46.4.0.
No hurries :)
Setuptools_scm and other setuptools plugins are not intended to be installed system-wide. Why not just not do that?
> does this relate to the keyword hook usage?

I'm not sure, but you can find out by testing with `setuptools<42` and see if the behavior is any different. It's possible that it is affected, because the `finalize_distribution_options` hook does run (and thus enumerate entry points) earlier than other hooks.
I had not ever seen that recommendation anywhere (I would be happy to see guidance to that effect).
I'm happy to take your word for it, but then 1) perhaps setuptools/pip could somehow recognize that (e.g. packages providing setuptools-related entry points) and indeed refuse to install them, and 2) should e.g. linux/conda packagers get a warning requesting them to remove the packages (https://packages.debian.org/sid/python3-setuptools-scm, https://www.archlinux.org/packages/community/any/python-setuptools-scm/, https://anaconda.org/conda-forge/setuptools_scm etc.)?
Now that I think about it, your best option is probably to use pip's build isolation to bypass system packages altogether. `pip --use-pep517 foo` should work.
PEP517 is not really an option for us because we need to support editable installs (because the package takes a while to compile).
By "we" I specifically meant Matplotlib, in this case; I was hoping to switch it from versioneer to setuptools_scm.
I'm able to replicate the issue with:

```
draft $ cat > setup.py
__import__('setuptools').setup(name='foo', setup_requires=['setuptools_scm>=4'])
draft $ pip-run -q 'setuptools_scm<4' -- setup.py --name
WARNING: The wheel package is not available.
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py:701: UserWarning: Module setuptools_scm was already imported from /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-0cfuhayx/setuptools_scm/__init__.py, but /Users/jaraco/draft/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
foo
```
Confirmed - the issue emerges at setuptools 42.

```
draft $ pip-run -q 'setuptools_scm<4' 'setuptools<42' -- setup.py --name
foo
draft $ pip-run -q 'setuptools_scm<4' 'setuptools==42' -- setup.py --name
WARNING: The wheel package is not available.
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-f_0y2mmd/setuptools/dist.py:724: UserWarning: Module setuptools_scm was already imported from /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-f_0y2mmd/setuptools_scm/__init__.py, but /Users/jaraco/draft/.eggs/setuptools_scm-4.1.1-py3.8.egg is being added to sys.path
  pkg_resources.working_set.add(dist, replace=True)
foo
```
I've figured out the issue. When `setup()` is called, it fires up a minimal `Distribution()` class, just enough to fetch the build requirements. It then adds those to the working set and proceeds to run `setup()` normally.

However, because constructing a `Distribution` triggers `finalize_distribution_options`, that causes plugins to get initialized and thus things like `setuptools_scm` to load before `setup_requires` was honored.

This patch works around the issue:

```
diff --git a/setuptools/__init__.py b/setuptools/__init__.py
index 811f3fd2..dbcb1723 100644
--- a/setuptools/__init__.py
+++ b/setuptools/__init__.py
@@ -128,7 +128,11 @@ if PY3:
 def _install_setup_requires(attrs):
     # Note: do not use `setuptools.Distribution` directly, as
     # our PEP 517 backend patch `distutils.core.Distribution`.
-    dist = distutils.core.Distribution(dict(
+    class NeuteredDistribution(distutils.core.Distribution):
+        def finalize_options(self):
+            pass
+
+    dist = NeuteredDistribution(dict(
         (k, v) for k, v in attrs.items()
         if k in ('dependency_links', 'setup_requires')
     ))
```
I've transferred the issue to the setuptools project, as setuptools_scm is just one plugin affected.

Resolving patch:
diff --git a/setuptools/__init__.py b/setuptools/__init__.py
--- a/setuptools/__init__.py
+++ b/setuptools/__init__.py
@@ -129,10 +129,27 @@ def _looks_like_package(path):
 def _install_setup_requires(attrs):
     # Note: do not use `setuptools.Distribution` directly, as
     # our PEP 517 backend patch `distutils.core.Distribution`.
-    dist = distutils.core.Distribution(dict(
-        (k, v) for k, v in attrs.items()
-        if k in ('dependency_links', 'setup_requires')
-    ))
+    class MinimalDistribution(distutils.core.Distribution):
+        """
+        A minimal version of a distribution for supporting the
+        fetch_build_eggs interface.
+        """
+        def __init__(self, attrs):
+            _incl = 'dependency_links', 'setup_requires'
+            filtered = {
+                k: attrs[k]
+                for k in set(_incl) & set(attrs)
+            }
+            distutils.core.Distribution.__init__(self, filtered)
+
+        def finalize_options(self):
+            """
+            Disable finalize_options to avoid building the working set.
+            Ref #2158.
+            """
+
+    dist = MinimalDistribution(attrs)
+
     # Honor setup.cfg's options.
     dist.parse_config_files(ignore_option_errors=True)
     if dist.setup_requires:


Test patch: