Repository: pypa/setuptools

Issue:
Debian-specific behavior lost when Setuptools adopts distutils
`pip install -e $PACKAGE_PATH` installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past.

`pip install $PACKAGE_PATH` continues to install to /usr/local/bin as expected.

Openstack downstream has temporarily capped setuptools until we know if this is expected behaviour or not.  Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015779.html for more context.


Discussion:
Thanks for the report. I'm initially unable to replicate the reported baseline behavior:

```
$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; python3 -m pip-run -q "setuptools<48" -- -m pip install -q --no-deps -e twine; ls /usr/bin/twine /usr/local/bin/twine'
ls: cannot access '/usr/local/bin/twine': No such file or directory
/usr/bin/twine
```

_note: [multipy-tox](/jaraco/multipy-tox) is Ubuntu Focal with Python 3.8.2 installed_

In that command, I use Docker to install `setuptools<48` to then run `pip install -e twine` (a local checkout), which runs yet no command exist as `/usr/local/bin/twine`, but the command does exist as `/usr/bin/twine`.

So it seems maybe the discriminating factor is not the Setuptools 48 release (though its change is plausibly implicated). In Setuptools 49.1, I've (temporarily) disabled the functionality introduced in Setuptools 48. Does that correct the issue (in your environment)?

What other factors could be causing this unexpected behavior, even with older Setuptools? How can I replicate the issue?
This shows the behavior:

```
sudo docker run -it --rm ubuntu:20.04 bash -c 'apt-get update && apt-get install -y python3-pip git && pip3 install -U pip && pip --version && pip install -U "setuptools<49" && git clone https://github.com/pypa/twine && pip install ./twine && which twine && pip uninstall -y twine && pip install -e ./twine && which twine'
```

Using setuptools 49.1.0 seems to fix it (the difference between the commands is the version of setuptools installed):

```
sudo docker run -it --rm ubuntu:20.04 bash -c 'apt-get update && apt-get install -y python3-pip git && pip3 install -U pip && pip --version && pip install -U "setuptools==49.1.0" && git clone https://github.com/pypa/twine && pip install ./twine && which twine && pip uninstall -y twine && pip install -e ./twine && which twine'
```

I won't bother to paste the output of the commands here because it is quite verbose, but if that would be useful I can add it as attachments I guess.
I see the mistake with my example above; I was using `pip-run` to temporarily override the version of setuptools, but `twine` uses PEP 517 to build, so the build isolation causes that value to be discarded. If I'm using `pip-run` to override build requirements, I should also pass `--no-build-isolation` to `pip install`.

I'm able to quickly replicate the issue with this command:

```
$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; python3 -m pip install -q "setuptools==48"; python3 -m pip install -q --no-deps -e twine; which twine'
/usr/bin/twine
```
With this command, I'm able to replicate the behavior natively in setuptools:

```
$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; cd twine; python3 -m pip install -q "setuptools==48"; python3 setup.py develop --no-deps; which twine'
running develop
running egg_info
creating twine.egg-info
writing twine.egg-info/PKG-INFO
writing dependency_links to twine.egg-info/dependency_links.txt
writing entry points to twine.egg-info/entry_points.txt
writing requirements to twine.egg-info/requires.txt
writing top-level names to twine.egg-info/top_level.txt
writing manifest file 'twine.egg-info/SOURCES.txt'
writing manifest file 'twine.egg-info/SOURCES.txt'
running build_ext
Creating /usr/lib/python3.8/site-packages/twine.egg-link (link to .)
Adding twine 3.2.1.dev1+g141428b to easy-install.pth file
Installing twine script to /usr/bin

Installed /twine
/usr/bin/twine
```
It seems that Debian and Ubuntu keep their own forks of distutils. I copied /usr/lib/python3.8/distutils over setuptools/_distutils and found diffs like this:

```diff
diff --git a/setuptools/_distutils/command/install.py b/setuptools/_distutils/command/install.py
index 13feeb89..2934862f 100644
--- a/setuptools/_distutils/command/install.py
+++ b/setuptools/_distutils/command/install.py
@@ -30,33 +30,33 @@ WINDOWS_SCHEME = {
 INSTALL_SCHEMES = {
     'unix_prefix': {
         'purelib': '$base/lib/python$py_version_short/site-packages',
-        'platlib': '$platbase/$platlibdir/python$py_version_short/site-packages',
+        'platlib': '$platbase/lib/python$py_version_short/site-packages',
         'headers': '$base/include/python$py_version_short$abiflags/$dist_name',
         'scripts': '$base/bin',
         'data'   : '$base',
         },
+    'unix_local': {
+        'purelib': '$base/local/lib/python$py_version_short/dist-packages',
+        'platlib': '$platbase/local/lib/python$py_version_short/dist-packages',
+        'headers': '$base/local/include/python$py_version_short/$dist_name',
+        'scripts': '$base/local/bin',
+        'data'   : '$base/local',
+        },
+    'deb_system': {
+        'purelib': '$base/lib/python3/dist-packages',
+        'platlib': '$platbase/lib/python3/dist-packages',
+        'headers': '$base/include/python$py_version_short/$dist_name',
+        'scripts': '$base/bin',
+        'data'   : '$base',
+        },
     'unix_home': {
         'purelib': '$base/lib/python',
-        'platlib': '$base/$platlibdir/python',
+        'platlib': '$base/lib/python',
         'headers': '$base/include/python/$dist_name',
         'scripts': '$base/bin',
         'data'   : '$base',
         },
     'nt': WINDOWS_SCHEME,
-    'pypy': {
-        'purelib': '$base/site-packages',
-        'platlib': '$base/site-packages',
-        'headers': '$base/include/$dist_name',
-        'scripts': '$base/bin',
-        'data'   : '$base',
-        },
-    'pypy_nt': {
-        'purelib': '$base/site-packages',
-        'platlib': '$base/site-packages',
-        'headers': '$base/include/$dist_name',
-        'scripts': '$base/Scripts',
-        'data'   : '$base',
-        },
     }
 
 # user site schemes
@@ -145,6 +145,9 @@ class install(Command):
 
         ('record=', None,
          "filename in which to record list of installed files"),
+
+        ('install-layout=', None,
+         "installation layout to choose (known values: deb, unix)"),
         ]
 
     boolean_options = ['compile', 'force', 'skip-build']
@@ -165,6 +168,7 @@ class install(Command):
         self.exec_prefix = None
         self.home = None
         self.user = 0
+        self.prefix_option = None
 
         # These select only the installation base; it's up to the user to
         # specify the installation scheme (currently, that means supplying
@@ -186,6 +190,10 @@ class install(Command):
         self.install_userbase = USER_BASE
         self.install_usersite = USER_SITE
 
+        # enable custom installation, known values: deb
+        self.install_layout = None
+        self.multiarch = None
+        
         self.compile = None
         self.optimize = None
 
@@ -312,7 +320,6 @@ class install(Command):
                             'sys_exec_prefix': exec_prefix,
                             'exec_prefix': exec_prefix,
                             'abiflags': abiflags,
-                            'platlibdir': getattr(sys, 'platlibdir', 'lib'),
                            }
 
         if HAS_USER_SITE:
@@ -428,6 +435,7 @@ class install(Command):
             self.install_base = self.install_platbase = self.home
             self.select_scheme("unix_home")
         else:
+            self.prefix_option = self.prefix
             if self.prefix is None:
                 if self.exec_prefix is not None:
                     raise DistutilsOptionError(
@@ -442,7 +450,28 @@ class install(Command):
 
             self.install_base = self.prefix
             self.install_platbase = self.exec_prefix
-            self.select_scheme("unix_prefix")
+            if self.install_layout:
+                if self.install_layout.lower() in ['deb']:
+                    import sysconfig
+                    self.multiarch = sysconfig.get_config_var('MULTIARCH')
+                    self.select_scheme("deb_system")
+                elif self.install_layout.lower() in ['unix']:
+                    self.select_scheme("unix_prefix")
+                else:
+                    raise DistutilsOptionError(
+                        "unknown value for --install-layout")
+            elif ((self.prefix_option and
+                   os.path.normpath(self.prefix) != '/usr/local')
+                  or sys.base_prefix != sys.prefix
+                  or 'PYTHONUSERBASE' in os.environ
+                  or 'VIRTUAL_ENV' in os.environ
+                  or 'real_prefix' in sys.__dict__):
+                self.select_scheme("unix_prefix")
+            else:
+                if os.path.normpath(self.prefix) == '/usr/local':
+                    self.prefix = self.exec_prefix = '/usr'
+                    self.install_base = self.install_platbase = '/usr'
+                self.select_scheme("unix_local")
 
     def finalize_other(self):
         """Finalizes options for non-posix platforms"""
@@ -469,12 +498,6 @@ class install(Command):
     def select_scheme(self, name):
         """Sets the install directories by applying the install schemes."""
         # it's the caller's problem if they supply a bad name!
-        if (hasattr(sys, 'pypy_version_info') and
-                not name.endswith(('_user', '_home'))):
-            if os.name == 'nt':
-                name = 'pypy_nt'
-            else:
-                name = 'pypy'
         scheme = INSTALL_SCHEMES[name]
         for key in SCHEME_KEYS:
             attrname = 'install_' + key

```

So when Setuptools adopts distutils from the standard library and uses it, any tweaks that downstream packagers have made to distutils will be lost.

Where are these tweaks managed? Can I get some help from Debian and/or Ubuntu to apply these tweaks to [distutils](/pypa/distutils)?
> Can I get some help from Debian and/or Ubuntu to apply these tweaks to [distutils](/pypa/distutils)?

/cc @kitterma (one of the Debian Python folks)
Also relevant: https://sources.debian.org/patches/python3.8/

Specifically: https://sources.debian.org/patches/python3.8/3.8.4%7Erc1-1/distutils-install-layout.diff/
I took a quick look at the patch and it looks pretty manageable. However, the options seem to change the default behavior. What we would need for distutils is something that behaves the same as currently by default, but on Debian honors the alternate behaviors.
Making the installation into /usr the default is definitely not appreciated.  I think we discussed this at length at one of the Cleveland sprints.

The behavior that you don't want to have is to install, remove, or update a package into /usr which was installed using the Debian package manager (apt/dpkg).  Afaics this is now exactly the thing which happens with the current setuptools behavior.  It's unfortunate that a change like this is again introduced. I don't think that the solution is to change any default for setuptools, but to make pip aware of dpkg/apt managed installations.
Here, Setuptools is only relying on the distutils behavior from CPython. All that's needed is to port the Debian-specific behavior into pypa/distutils and (especially if there's a test capturing the expectation), Setuptools will ever-after support that behavior without the need for Debian to patch the software when installing it. If the Debian behavior is preferable for more environments, the project can consider employing it there as well.

To be certain, this change was not expected and was disabled within 24 hours. Now all we're seeking is to settle on a long-term solution in the pypa/distutils repo that can become the reference implementation without any downstream patches.
>  I think we discussed this at length at one of the Cleveland sprints.

By the way, I remember the discussions and enjoyed our exchange. Unfortunately, so much time has passed that I haven't retained any actionable points from our discussion.

Do you have a recommendation? I don't think the diverse community of Setuptools users can readily adopt the Debian patch wholesale. Perhaps Setuptools/distutils could adopt those changes over time, giving other operating systems time to adapt to the approach Debian uses... or maybe those platforms have sound reasons for not having those changes. In any case, I believe the near-term solution, to unblock Setuptools from adopting distutils, would be for pypa/distutils to adopt the Debian patch but enable the Debian default settings only on Debian. Can you advise how to reliably detect hosts that would opt into this behavior?
> Unfortunately, so much time has passed that I haven't retained any actionable points from our discussion.

If I remember correctly...

The conclusion was that Debian and other Linux distros should stop shipping RECORD in dist-info folders for packages that they want to be managing (which renders pip unable to uninstall them, and thus won't do anything with them). This was a quick-fix-that-works-now and would have a suboptimal error message at the start which could be changed a bit in future releases and the change would be backported by the distros.

Another, longer-term approach for presenting a better error message would be to have some agreement on how to indicate a package as "externally managed" in the metadata. I'd initially proposed just using INSTALLER, but folks did flag that "externally managed" is different piece of metadata than "end user installer" (pip/pipenv/dpkg are valid INSTALLER values, but only 1 is externally managed, and pip shouldn't be keeping an allowlist of any kind) and that we just need to augment the information - either yet-another file or changing the contents of installer.

I do remember that someone from the Debian team did start working on a patch to update all the Debian packages to remove RECORD files at the Sprints, but idk if this ever rolled out. Some folks (not me!) did volunteer to start a discussion / PEP for the "externally managed" meradata. I don't think I ever saw a PEP draft for this, but I could be wrong.

Oh, and for pip, adding an INSTALLER file that didn't contain "pip" was a cleaner patch than manipulating the code to disable the version check.

We needed these details to be documented in some form, and IIRC, one of the Linux distro folks (or me?) did volunteer to file a PR but I don't think that ever happened. 
This was of course, in the direction of distros stopping patches like debian distutils patch, in favor of better metadata to indicate these details and to have behaviors originating from upstream (like pip's pypa/pip#7002) that remove the likelihood of users getting PermissionError, overwritten files and things like that.

Please have a look at #2240, which attempts to address #2232 by integrating the Debian patch and enabling its default behaviors on Debian only.
I don't like the idea of having "Debian only" code upstream.  Would it be possible to add options instead, like --unversioned-eggs to do the same in a more generic way?

In #2240, there seems to be some consensus that incorporating the Debian patches as debian-only is undesirable.

Here are the options I see:

- Simply re-enable the functionality with the known degradation for Debian systems. Let Debian maintainers patch Setuptools where possible but otherwise Debian users get the general/unpatched behavior. Possibly enhance the behavior later to satisfy Debian needs in a generic way.
- Defer enabling distutils adoption until a satisfactory solution can be implemented that supports Debian's needs and Debian is prepared to enable that functionality.
- Accept the PR to incorporate Debian-specific behavior and feature parity with Debian-patched distutils with an understanding that Debian maintainers will help develop a more permanent, general solution ASAP.

Thoughts? Other options?
I may not fully understand the scope of who is affected, but my vote is for the first option. Debian (and other Debian-like systems) should be able to pin their adoption of setuptools until they have patches in place, and it's best to keep the roll-out moving so people on other distros can find their own regressions 😅.
Without more feedback, Setuptools is very likely to re-release the behavior that led to this report with the recommendation for Debian users to pin to older Setuptools. I'm open to arguments or suggestions on how to smooth the transition for Debian users. If there's another person or forum for engaging on this issue, please direct us there.
I'm generally in favor of option 1, although I want our primary intent to be to work with Debian maintainers to improve the situation ASAP.

With my limited understanding of the history here (and nuances here due to it), I think a lot of the recent changes in Python Packaging tooling have made a lot of debian's patches to Python packaging tools in general to be no-longer-necessary (or at least very-defensive) and I am generally in favor of Linux distros dropping their patches on Python Packaging tooling, and collaborating with us to come up with more general upstream solutions instead.

> Without more feedback, Setuptools is very likely to re-release the behavior that led to this report with the recommendation for Debian users to pin to older Setuptools. I'm open to arguments or suggestions on how to smooth the transition for Debian users. If there's another person or forum for engaging on this issue, please direct us there.

Is it a problem that `pip install` and `pip install -e` will install to different path prefixes? Consistency there would be good even if the path prefix ultimately changes.
> Is it a problem that `pip install` and `pip install -e` will install to different path prefixes? Consistency there would be good even if the path prefix ultimately changes.

I agree it would be nice if there was consistency and maybe there's a deficiency in Setuptools that could be addressed here.

I don't have a grasp of the design here, or even what is intentionally designed versus a defacto approach that's evolved over time.

I do see that [pip relies on stdlib distutils](https://github.com/pypa/pip/blob/8bf5731b84907bd1b41846334e98530bbe0cceaf/src/pip/_internal/locations.py#L95-L156) at least in some scenarios.

And in fact, if you enable the distutils patch by way of importing setuptools before invoking pip, pip also exhibits the same behavior:

```
$ docker run -it jaraco/multipy-tox sh -c 'python -m pip-run -q setuptools==49.0.1 -- -c "import setuptools; from pip._internal.cli.main import main; main()" install -q twine && which twine'
/usr/bin/twine
```

So it does seem that when [distutils is deprecated](https://bugs.python.org/issue41282) in the standard library and pip begins to use distutils as supplied by either pypa/distutils or Setuptools, pip also will begin installing to `/usr/bin` on Debian. So the behavior is theoretically consistent; it's just out of sync.

I found pypa/virtualenv#1770 which has some good leads for expanding the visibility of this issue and with any luck to garner more support around addressing the issue properly. I plan to reach out on IRC to let others know this change is coming.

I'm going to draft the PR for restoring distutils adoption by default, which will re-apply this undesirable effect for editable installs in Debian, but I'm hopeful that a long term solution can be developed against pypa/distutils#2.

Resolving patch:
diff --git a/_distutils_hack/__init__.py b/_distutils_hack/__init__.py
--- a/_distutils_hack/__init__.py
+++ b/_distutils_hack/__init__.py
@@ -37,7 +37,7 @@ def enabled():
     """
     Allow selection of distutils by environment variable.
     """
-    which = os.environ.get('SETUPTOOLS_USE_DISTUTILS', 'stdlib')
+    which = os.environ.get('SETUPTOOLS_USE_DISTUTILS', 'local')
     return which == 'local'
 
 
@@ -66,12 +66,14 @@ def do_override():
 
 class DistutilsMetaFinder:
     def find_spec(self, fullname, path, target=None):
-        if path is not None or fullname != "distutils":
-            return None
+        if path is not None:
+            return
 
-        return self.get_distutils_spec()
+        method_name = 'spec_for_{fullname}'.format(**locals())
+        method = getattr(self, method_name, lambda: None)
+        return method()
 
-    def get_distutils_spec(self):
+    def spec_for_distutils(self):
         import importlib.util
 
         class DistutilsLoader(importlib.util.abc.Loader):
@@ -84,6 +86,14 @@ def exec_module(self, module):
 
         return importlib.util.spec_from_loader('distutils', DistutilsLoader())
 
+    def spec_for_pip(self):
+        """
+        Ensure stdlib distutils when running under pip.
+        See pypa/pip#8761 for rationale.
+        """
+        clear_distutils()
+        self.spec_for_distutils = lambda: None
+
 
 DISTUTILS_FINDER = DistutilsMetaFinder()
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -99,7 +99,8 @@ class install_with_pth(install):
     _pth_name = 'distutils-precedence'
     _pth_contents = textwrap.dedent("""
         import os
-        enabled = os.environ.get('SETUPTOOLS_USE_DISTUTILS') == 'local'
+        var = 'SETUPTOOLS_USE_DISTUTILS'
+        enabled = os.environ.get(var, 'local') == 'local'
         enabled and __import__('_distutils_hack').add_shim()
         """).lstrip().replace('\n', '; ')
 


Test patch:
diff --git a/setuptools/tests/test_distutils_adoption.py b/setuptools/tests/test_distutils_adoption.py
--- a/setuptools/tests/test_distutils_adoption.py
+++ b/setuptools/tests/test_distutils_adoption.py
@@ -48,15 +48,15 @@ def test_distutils_stdlib(venv):
     """
     Ensure stdlib distutils is used when appropriate.
     """
-    assert venv.name not in find_distutils(venv, env=dict()).split(os.sep)
+    env = dict(SETUPTOOLS_USE_DISTUTILS='stdlib')
+    assert venv.name not in find_distutils(venv, env=env).split(os.sep)
 
 
 def test_distutils_local_with_setuptools(venv):
     """
     Ensure local distutils is used when appropriate.
     """
-    env = dict(SETUPTOOLS_USE_DISTUTILS='local')
-    loc = find_distutils(venv, imports='setuptools, distutils', env=env)
+    loc = find_distutils(venv, imports='setuptools, distutils', env=dict())
     assert venv.name in loc.split(os.sep)
 
 
@@ -66,5 +66,4 @@ def test_distutils_local(venv):
     Even without importing, the setuptools-local copy of distutils is
     preferred.
     """
-    env = dict(SETUPTOOLS_USE_DISTUTILS='local')
-    assert venv.name in find_distutils(venv, env=env).split(os.sep)
+    assert venv.name in find_distutils(venv, env=dict()).split(os.sep)