Repository: pypa/setuptools

Issue:
Unexpected behavior when distutils is invoked before Setuptools
TL;DR We are forced to use distutils because setuptools has broken symlink processing and this causes `distutils.errors.DistutilsClassError: command class <class '__main__.SDistCommand'> must subclass Command`.

It works with `setuptools<48` and the changelog doesn't document any breaking behaviors for this version.

Repro:
```console
$ git clone https://github.com/ansible/ansible.git
$ cd ansible
$ pip install -U 'setuptools>=48'
$ python setup.py sdist
```
(tried under Python 3.8)

Ref: https://github.com/ansible/ansible/issues/70456


Discussion:
Thanks for the report.

Setuptools 49.1 is out, should address the issue for early adopters while I triage and correct the issue.
I'm able to replicate the failure:

```
ansible devel $ pip-run -q setuptools==48 -- setup.py sdist
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/distutils_patch.py:17: UserWarning: Setuptools is replacing distutils
  warnings.warn("Setuptools is replacing distutils")
Traceback (most recent call last):
  File "setup.py", line 338, in <module>
    main()
  File "setup.py", line 333, in main
    setup(**setup_params)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/__init__.py", line 164, in setup
    return distutils.core.setup(**attrs)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/core.py", line 134, in setup
    ok = dist.parse_command_line()
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/dist.py", line 484, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/dist.py", line 925, in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/dist.py", line 547, in _parse_command_opts
    raise DistutilsClassError(
distutils.errors.DistutilsClassError: command class <class '__main__.SDistCommand'> must subclass Command
```

Note the warning "Setuptools is replacing distutils".

That was meant to be a hard failure, except that PyPy somehow ends up with distutils unconditionally imported during interpreter startup.
I'm able to work around the issue by ensuring that `setuptools` is imported prior to running the command:

```
$ pip-run -q setuptools==48 -- -c "import setuptools; exec(open('setup.py').read())" sdist
...
hard linking test/units/vars/test_variable_manager.py -> ansible-base-2.11.0.dev0/test/units/vars
creating dist
Creating tar archive
removing 'ansible-base-2.11.0.dev0' (and everything under it)
<string>:180: RuntimeWarning: When setup.py sdist is run from outside of the Makefile, the generated tarball may be incomplete.  Use `make snapshot` to create a tarball from an arbitrary checkout or use `cd packaging/release && make release version=[..]` for official builds.
```
Also using PEP 517:

```
ansible devel $ cat > pyproject.toml
[build-system]
requires = ['setuptools', 'wheel']
build-backend = 'setuptools.build_meta'

ansible devel $ pip-run -q pep517 -- -m pep517.build -s .
running egg_info
writing lib/ansible_base.egg-info/PKG-INFO
writing dependency_links to lib/ansible_base.egg-info/dependency_links.txt
writing requirements to lib/ansible_base.egg-info/requires.txt
writing top-level names to lib/ansible_base.egg-info/top_level.txt
reading manifest file 'lib/ansible_base.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'docs/docsite/rst_warnings'
warning: no previously-included files matching '*' found under directory 'docs/docsite/_build'
warning: no previously-included files matching '*.pyc' found under directory 'docs/docsite/_extensions'
warning: no previously-included files matching '*.pyo' found under directory 'docs/docsite/_extensions'
warning: no files found matching '*.ps1' under directory 'lib/ansible/modules/windows'
warning: no files found matching '*.psm1' under directory 'test/support'
writing manifest file 'lib/ansible_base.egg-info/SOURCES.txt'
running sdist
...
Creating tar archive
removing 'ansible-base-2.11.0.dev0' (and everything under it)
setup.py:180: RuntimeWarning: When setup.py sdist is run from outside of the Makefile, the generated tarball may be incomplete.  Use `make snapshot` to create a tarball from an arbitrary checkout or use `cd packaging/release && make release version=[..]` for official builds.
  warnings.warn('When setup.py sdist is run from outside of the Makefile,'
```
It also works with this diff on ansible:

```diff
diff --git a/setup.py b/setup.py
index 0398473d5e..7c86075495 100644
--- a/setup.py
+++ b/setup.py
@@ -9,8 +9,6 @@ import sys
 import warnings
 
 from collections import defaultdict
-from distutils.command.build_scripts import build_scripts as BuildScripts
-from distutils.command.sdist import sdist as SDist
 
 try:
     from setuptools import setup, find_packages
@@ -23,6 +21,9 @@ except ImportError:
           " install setuptools).", file=sys.stderr)
     sys.exit(1)
 
+from distutils.command.build_scripts import build_scripts as BuildScripts
+from distutils.command.sdist import sdist as SDist
+
 sys.path.insert(0, os.path.abspath('lib'))
 from ansible.release import __version__, __author__
```

The underlying issue is that with setuptools 48, you cannot import distutils before setuptools.

My recommendation is to (a) move the imports so distutils imports come after setuptools (quick, painless, compatible), then (b) adopt PEP 517 and use [pep517](https://pypi.org/project/pep517) to build your sdists.

I'll update the changelog to make this change more clear.
In the referenced commit, I've updated the changelog to clarify the weaknesses identified from this bug report and guide users to the best practices. Can you confirm that this guidance and applying the recommended changes in Ansible addresses the concern?
> Can you confirm that this guidance and applying the recommended changes in Ansible addresses the concern?

I've submitted the patch that you suggested and it'll probably get us going short-term. But we cannot use pyproject.toml atm, nor we ship wheels. The reason is that we have a bunch of symlinks that need to be preserved and it's done by distutils but not setuptools.
Great. Thanks for enacting that.

I'm wondering if there's something that Setuptools should do here to help other projects with similar usages from encountering the same issue as we bring the adopted distutils back. For example, the project could:

1. Prior to re-enabling the adoption, warn when 'distutils' appears in `sys.modules` (except on PyPy, where that's the status quo on startup).
1. When re-enabling the adoption, error with a helpful message when distutils is replaced (again exempting PyPy).

I think I'll enact (1) now and give it some time to percolate over the weekend and maybe longer.

Do please consider filing something with Ansible to investigate a long-term solution for what Ansible will do when distutils is sunset and all that's left is Setuptools (as that's the plan).

Resolving patch:
diff --git a/setuptools/distutils_patch.py b/setuptools/distutils_patch.py
--- a/setuptools/distutils_patch.py
+++ b/setuptools/distutils_patch.py
@@ -12,10 +12,26 @@
 import warnings
 
 
+is_pypy = '__pypy__' in sys.builtin_module_names
+
+
+def warn_distutils_present():
+    if 'distutils' not in sys.modules:
+        return
+    if is_pypy and sys.version_info < (3, 7):
+        # PyPy for 3.6 unconditionally imports distutils, so bypass the warning
+        # https://foss.heptapod.net/pypy/pypy/-/blob/be829135bc0d758997b3566062999ee8b23872b4/lib-python/3/site.py#L250
+        return
+    warnings.warn(
+        "Distutils was imported before Setuptools. This usage is discouraged "
+        "and may exhibit undesirable behaviors or errors. Please use "
+        "Setuptools' objects directly or at least import Setuptools first.")
+
+
 def clear_distutils():
     if 'distutils' not in sys.modules:
         return
-    warnings.warn("Setuptools is replacing distutils")
+    warnings.warn("Setuptools is replacing distutils.")
     mods = [name for name in sys.modules if re.match(r'distutils\b', name)]
     for name in mods:
         del sys.modules[name]
@@ -40,5 +56,6 @@ def ensure_local_distutils():
     assert '_distutils' in core.__file__, core.__file__
 
 
+warn_distutils_present()
 if enabled():
     ensure_local_distutils()


Test patch: