Repository: pypa/setuptools

Issue:
[BUG] setuptools 58.5 breaks installation of docutils==0.10.0
### setuptools version

58.5.{0,1}

### Python version

3.9

### OS

Arch Linux

### Additional environment information

_No response_

### Description

Installing docutils==0.10 in a clean venv with setuptools 58.5.{0,1} fails with an internal(?) AttributeError.

### Expected behavior

Successful installation.

### How to Reproduce

```sh
python -mvenv /tmp/testenv && source /tmp/testenv/bin/activate &&
  pip install -U pip setuptools wheel && pip install docutils==0.10
```
(To repro the issue, you may need to `pip cache purge` first if you happen to already have a wheel for docutils==0.10 in your pip cache.)

### Output

```console
  ERROR: Command errored out with exit status 1:
   command: /tmp/testenv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/setup.py'"'"'; __file__='"'"'/tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-_qz_txff
       cwd: /tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/
  Complete output (35 lines):
  running egg_info
  creating /tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info
  writing /tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info/SOURCES.txt'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/setup.py", line 221, in <module>
      do_setup()
    File "/tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/setup.py", line 98, in do_setup
      dist = setup(**kwargs)
    File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 299, in run
      self.find_sources()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
      mm.run()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 541, in run
      self.add_defaults()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
      sdist.add_defaults(self)
    File "/usr/lib/python3.9/distutils/command/sdist.py", line 226, in add_defaults
      self._add_defaults_python()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/sdist.py", line 113, in _add_defaults_python
      self._add_data_files(self._safe_data_files(build_py))
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 621, in _safe_data_files
      return build_py.get_data_files_without_manifest()
    File "/usr/lib/python3.9/distutils/cmd.py", line 103, in __getattr__
      raise AttributeError(attr)
  AttributeError: get_data_files_without_manifest
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/66/40/2b11ec2ec96d527ef933332a5847cc90860fd8bbee46da2cc468794ea25b/docutils-0.10.tar.gz#sha256=370624e61b6773da2f2fb17cc2a4eaea4bb596c3585d13f75ff193c1c738603e (from https://pypi.org/simple/docutils/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement docutils==0.10 (from versions: 0.3, 0.3.5, 0.3.7, 0.3.9, 0.4, 0.5, 0.6, 0.7, 0.8, 0.8.1, 0.9, 0.9.1, 0.10, 0.11, 0.12, 0.13.1, 0.14rc1, 0.14rc2, 0.14, 0.15, 0.15.1.post1, 0.15.2, 0.16b0.dev0, 0.16rc1, 0.16, 0.17b1, 0.17, 0.17.1b1.dev0, 0.17.1, 0.18b1, 0.18)
ERROR: No matching distribution found for docutils==0.10
```


### Code of Conduct

- [X] I agree to follow the PSF Code of Conduct


Discussion:
breaks pycrypto too
Breaks `python setup.py bdist_wheel -d build` [in our pipeline](https://dev.azure.com/azure-sdk/public/_build/results?buildId=1178162&view=logs&jobId=e725b3e9-648f-510f-a0a1-f18bdc151d12&j=e725b3e9-648f-510f-a0a1-f18bdc151d12&t=7dda8e6e-5520-5cbf-e881-69c151aecb23):

<details>
<summary>Full call stack</summary>

```
2021-11-04T03:23:41.8710919Z Traceback (most recent call last):
2021-11-04T03:23:41.8711856Z   File "/home/vsts/work/1/s/src/azure-cli/setup.py", line 160, in <module>
2021-11-04T03:23:41.8712337Z     setup(
2021-11-04T03:23:41.8713101Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/__init__.py", line 159, in setup
2021-11-04T03:23:41.8713849Z     return distutils.core.setup(**attrs)
2021-11-04T03:23:41.8714381Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/core.py", line 148, in setup
2021-11-04T03:23:42.0792338Z     dist.run_commands()
2021-11-04T03:23:42.0793141Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 966, in run_commands
2021-11-04T03:23:42.1846501Z     self.run_command(cmd)
2021-11-04T03:23:42.1847638Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 985, in run_command
2021-11-04T03:23:42.1848465Z     cmd_obj.run()
2021-11-04T03:23:42.1849964Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 335, in run
2021-11-04T03:23:42.1850889Z     self.run_command('install')
2021-11-04T03:23:42.1851397Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/cmd.py", line 313, in run_command
2021-11-04T03:23:42.2316182Z     self.distribution.run_command(command)
2021-11-04T03:23:42.2317514Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 985, in run_command
2021-11-04T03:23:42.2320510Z     cmd_obj.run()
2021-11-04T03:23:42.2321832Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/install.py", line 68, in run
2021-11-04T03:23:42.2322583Z     return orig.install.run(self)
2021-11-04T03:23:42.2323717Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/command/install.py", line 558, in run
2021-11-04T03:23:42.2335935Z     self.run_command(cmd_name)
2021-11-04T03:23:42.2336648Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/cmd.py", line 313, in run_command
2021-11-04T03:23:42.2337157Z     self.distribution.run_command(command)
2021-11-04T03:23:42.2337636Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 985, in run_command
2021-11-04T03:23:42.2338089Z     cmd_obj.run()
2021-11-04T03:23:42.2349505Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/install_egg_info.py", line 34, in run
2021-11-04T03:23:42.2350417Z     self.run_command('egg_info')
2021-11-04T03:23:42.2350922Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/cmd.py", line 313, in run_command
2021-11-04T03:23:42.2351403Z     self.distribution.run_command(command)
2021-11-04T03:23:42.2351893Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 985, in run_command
2021-11-04T03:23:42.2357369Z     cmd_obj.run()
2021-11-04T03:23:42.2358817Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 299, in run
2021-11-04T03:23:42.2359947Z     self.find_sources()
2021-11-04T03:23:42.2361214Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
2021-11-04T03:23:42.2362230Z     mm.run()
2021-11-04T03:23:42.2363085Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 541, in run
2021-11-04T03:23:42.2364084Z     self.add_defaults()
2021-11-04T03:23:42.2364807Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
2021-11-04T03:23:42.2365253Z     sdist.add_defaults(self)
2021-11-04T03:23:42.2365673Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/command/sdist.py", line 226, in add_defaults
2021-11-04T03:23:42.2366081Z     self._add_defaults_python()
2021-11-04T03:23:42.2366796Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/sdist.py", line 113, in _add_defaults_python
2021-11-04T03:23:42.2367678Z     self._add_data_files(self._safe_data_files(build_py))
```

</details>

```
2021-11-04T03:23:42.2368540Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 621, in _safe_data_files
2021-11-04T03:23:42.2369747Z     return build_py.get_data_files_without_manifest()
2021-11-04T03:23:42.2370277Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/cmd.py", line 103, in __getattr__
2021-11-04T03:23:42.2370751Z     raise AttributeError(attr)
2021-11-04T03:23:42.2374899Z AttributeError: get_data_files_without_manifest
```


`get_data_files_without_manifest` was added by https://github.com/pypa/setuptools/pull/2844

This seems to be related to the replacement of https://github.com/pypa/setuptools/commit/8a0e63a586674676fa13b0ee4cfde46228f8a266 with https://github.com/pypa/setuptools/commit/b6ce5368a439ad55cd1ada777676f387be895946.

Those entry-points should point to the setuptools version of the commands, but for some reason, the original distutils commands are being used instead.

---

Building the project with `build` seems to work fine, though:

```bash
$ virtualenv venv-docutils
$ source venv-docutils/bin/activate
$ pip cache purge
$ svn checkout https://svn.code.sf.net/p/docutils/code/trunk/docutils docutils-code
$ cd docutils-code
$ pip install build
$ python -m build
```
This appears to be a breaking change introduced in https://github.com/pypa/setuptools/commit/2e66eb7147ae1e8a799b2276c7340f48f63a7220 when custom commands are used as it forces the new function `get_data_files_without_manifest` to be present.

> This appears to be a breaking change introduced in [2e66eb7](https://github.com/pypa/setuptools/commit/2e66eb7147ae1e8a799b2276c7340f48f63a7220) when custom commands are used as it forces the new function `get_data_files_without_manifest` to be present.

I confirm that pip installing using 1812f8db08e32c72b4e1e419ad178ed0e66b2958 works but using 2e66eb7 fails (tested on scikit-learn).
Also breaks install for Mercurial: https://foss.heptapod.net/mercurial/mercurial-devel/-/jobs/260776#L53
I think we have 2 separated problems here:

1. If packages define their own `build_py` inheriting from `distutils` instead of setuptools, and don't define a `get_data_files_without_manifest`, there will be a problem.

2. Installing `sdist` packages with pip might result in error, since the `distutils` version of `build_py` is being used instead of the `setuptools` version.
   This can be reproduced with `docutils==0.10.0`.
   Interestingly, `pypa/build` isolated environments don't seem to have the same problem.
   This second issue seems more difficult to track down.

---

Correction, this is indeed a single problem, as pointed out by @nshgraph

`docutils==0.10.0` defines a custom version of `copy_build_py_2to3` that inherits from `distutils.command.build_py.build_py_2to3`.

The latest version of `docutils` do not have this requirement. @anntzer are you unable to update the version of `docutils`?
I certainly can, this was just a version I had pinned in CI to test an "oldest-supported-version-of-dependency" for another package.  I guess the other commenters here may want a more robust solution, though...
> I certainly can, this was just a version I had pinned in CI to test an "oldest-supported-version-of-dependency" for another package. I guess the other commenters here may want a more robust solution, though...

Yes, I understand that.

I am having a look to try to identify what can be done:

- Some packages listed here use `build_py_2to3` (e.g. docutils, pycrypto)
- Others define a custom `build_py` inheriting directly from `distutils` (e.g. pycryptodome)
- There is a third case that I am still trying to find a connection. This is the case of packages that redefine `build_ext`, `sdist` or custom builds but not `build_py`.


---
@jaraco maybe the easiest approach is to restore https://github.com/pypa/setuptools/commit/8a0e63a586674676fa13b0ee4cfde46228f8a266?

Unless supporting commands that inherit directly from `distutils` is outside of the scope of setuptools, which in that case would require adding a "breaking change" notice to the change log.

Resolving patch:
diff --git a/setuptools/command/egg_info.py b/setuptools/command/egg_info.py
--- a/setuptools/command/egg_info.py
+++ b/setuptools/command/egg_info.py
@@ -618,7 +618,15 @@ def _safe_data_files(self, build_py):
         Therefore, avoid triggering any attempt of
         analyzing/building the manifest again.
         """
-        return build_py.get_data_files_without_manifest()
+        if hasattr(build_py, 'get_data_files_without_manifest'):
+            return build_py.get_data_files_without_manifest()
+
+        log.warn(
+            "Custom 'build_py' does not implement "
+            "'get_data_files_without_manifest'.\nPlease extend command classes"
+            " from setuptools instead of distutils."
+        )
+        return build_py.get_data_files()
 
 
 def write_file(filename, contents):


Test patch:
diff --git a/setuptools/tests/test_sdist.py b/setuptools/tests/test_sdist.py
--- a/setuptools/tests/test_sdist.py
+++ b/setuptools/tests/test_sdist.py
@@ -6,6 +6,7 @@
 import unicodedata
 import contextlib
 import io
+from unittest import mock
 
 import pytest
 
@@ -106,6 +107,13 @@ def source_dir(self, tmpdir):
         with tmpdir.as_cwd():
             yield
 
+    def assert_package_data_in_manifest(self, cmd):
+        manifest = cmd.filelist.files
+        assert os.path.join('sdist_test', 'a.txt') in manifest
+        assert os.path.join('sdist_test', 'b.txt') in manifest
+        assert os.path.join('sdist_test', 'c.rst') not in manifest
+        assert os.path.join('d', 'e.dat') in manifest
+
     def test_package_data_in_sdist(self):
         """Regression test for pull request #4: ensures that files listed in
         package_data are included in the manifest even if they're not added to
@@ -120,11 +128,7 @@ def test_package_data_in_sdist(self):
         with quiet():
             cmd.run()
 
-        manifest = cmd.filelist.files
-        assert os.path.join('sdist_test', 'a.txt') in manifest
-        assert os.path.join('sdist_test', 'b.txt') in manifest
-        assert os.path.join('sdist_test', 'c.rst') not in manifest
-        assert os.path.join('d', 'e.dat') in manifest
+        self.assert_package_data_in_manifest(cmd)
 
     def test_package_data_and_include_package_data_in_sdist(self):
         """
@@ -142,11 +146,58 @@ def test_package_data_and_include_package_data_in_sdist(self):
         with quiet():
             cmd.run()
 
-        manifest = cmd.filelist.files
-        assert os.path.join('sdist_test', 'a.txt') in manifest
-        assert os.path.join('sdist_test', 'b.txt') in manifest
-        assert os.path.join('sdist_test', 'c.rst') not in manifest
-        assert os.path.join('d', 'e.dat') in manifest
+        self.assert_package_data_in_manifest(cmd)
+
+    @mock.patch('setuptools.command.egg_info.log')
+    def test_custom_build_py(self, log_stub):
+        """
+        Ensure projects defining custom build_py don't break
+        when creating sdists (issue #2849)
+        """
+        from distutils.command.build_py import build_py as OrigBuildPy
+
+        using_custom_command_guard = mock.Mock()
+
+        class CustomBuildPy(OrigBuildPy):
+            """
+            Some projects have custom commands inheriting from `distutils`
+            """
+
+            def get_data_files(self):
+                using_custom_command_guard()
+                return super().get_data_files()
+
+        setup_attrs = {**SETUP_ATTRS, 'include_package_data': True}
+        assert setup_attrs['package_data']
+
+        dist = Distribution(setup_attrs)
+        dist.script_name = 'setup.py'
+        cmd = sdist(dist)
+        cmd.ensure_finalized()
+
+        # Make sure we use the custom command
+        cmd.cmdclass = {'build_py': CustomBuildPy}
+        cmd.distribution.cmdclass = {'build_py': CustomBuildPy}
+        assert cmd.distribution.get_command_class('build_py') == CustomBuildPy
+
+        with quiet():
+            cmd.run()
+
+        using_custom_command_guard.assert_called()
+        self.assert_package_data_in_manifest(cmd)
+
+        warn_stub = log_stub.warn
+        warn_stub.assert_called()
+        for call in warn_stub.call_args_list:
+            args, _kw = call
+            if "setuptools instead of distutils" in args[0]:
+                return
+        else:
+            raise AssertionError(
+                "The user should have been warned to extend setuptools command"
+                " classes instead of distutils",
+                warn_stub.call_args_list
+            )
 
     def test_setup_py_exists(self):
         dist = Distribution(SETUP_ATTRS)