Repository: pypa/setuptools

Issue:
pip install -e fails on version 50: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command
```
> pip install -e .
    ERROR: Command errored out with exit status 1:
     command: /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/jaykarimi/Documents/vanir/setup.py'"'"'; __file__='"'"'/Users/jaykarimi/Documents/vanir/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /Users/jaykarimi/Documents/vanir/
    Complete output (19 lines):
    /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
      warnings.warn("Setuptools is replacing distutils.")
    /Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing 'v1.7.1' to '1.7.1'
      warnings.warn(tmpl.format(**locals()))
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/jaykarimi/Documents/vanir/setup.py", line 127, in <module>
        setup(**_conf)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/core.py", line 134, in setup
        ok = dist.parse_command_line()
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/dist.py", line 484, in parse_command_line
        args = self._parse_command_opts(parser, args)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/dist.py", line 903, in _parse_command_opts
        nargs = _Distribution._parse_command_opts(self, parser, args)
      File "/Users/jaykarimi/.pyenv/versions/3.6.5/envs/ci-debug/lib/python3.6/site-packages/setuptools/_distutils/dist.py", line 548, in _parse_command_opts
        "command class %s must subclass Command" % cmd_class)
    distutils.errors.DistutilsClassError: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
```


Discussion:
Minimal reproducing example setup.py
```
import setuptools
import pip
setuptools.setup()
```
So seems related to https://github.com/pypa/setuptools/issues/2365, though the symptoms are different
Since v50 we also get this error when installing open source package `doit` from its TGZ package (logs from a Windows machine, same error on Linux):

```
(venv) λ pip --no-cache-dir install doit
Looking in indexes: https://artifactory.cc.bmwgroup.net/artifactory/api/pypi/ic-gen5-pypi/simple
Collecting doit
  Downloading https://artifactory.cc.bmwgroup.net/artifactory/api/pypi/ic-gen5-pypi/packages/packages/1f/89/b5e1dd4854b47c4f9e5250899c04e0cd0c9082c30ef619c0413ace20cf7a/doit-0.32.0.tar.gz (1.4 MB)
     |████████████████████████████████| 1.4 MB 1.7 MB/s
    ERROR: Command errored out with exit status 1:
     command: 'd:\temp\doit-install-issue\venv\scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\q199925\\AppData\\Local\\Temp\\pip-install-by7x_6ei\\doit\\setup.py'"'"'; __file__='"'"'C:\\Users\\q199925\\AppData\\Local\\Temp\\pip-install-by7x_6ei\\doit\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\q199925\AppData\Local\Temp\pip-pip-egg-info-fhnoqhon'
         cwd: C:\Users\q199925\AppData\Local\Temp\pip-install-by7x_6ei\doit\
    Complete output (17 lines):
    d:\temp\doit-install-issue\venv\lib\site-packages\_distutils_hack\__init__.py:30: UserWarning: Setuptools is replacing distutils.
      warnings.warn("Setuptools is replacing distutils.")
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\q199925\AppData\Local\Temp\pip-install-by7x_6ei\doit\setup.py", line 44, in <module>
        setup(name = 'doit',
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\__init__.py", line 152, in setup
        _install_setup_requires(attrs)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\__init__.py", line 145, in _install_setup_requires
        dist.parse_config_files(ignore_option_errors=True)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\dist.py", line 665, in parse_config_files
        self._parse_config_files(filenames=filenames)
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\dist.py", line 572, in _parse_config_files
        filenames = self.find_config_files()
      File "d:\temp\doit-install-issue\venv\lib\site-packages\setuptools\_distutils\dist.py", line 353, in find_config_files
        sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
    KeyError: 'distutils'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
```

It also has an `import pip` in its `setup.py`. Please note that while this might be wrong, it's quite tough to force other projects to adapt by releasing a new version of setuptools.

Also, please note that [official Python docs](https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date) recommend to update `setuptools` to the latest version _before_ installing anything for real, so pinning `setuptools` to overcome the current issues is not a real option.

Would it be possible to roll back, but release a new version such that all the `pip install -U setuptools` calls remain valid?

@ferraith FYI
This can be worked around with the SETUPTOOLS_USE_DISTUTILS env var.
https://setuptools.readthedocs.io/en/latest/history.html#v50-0-0

I had this problem with `ruamel.yaml.clib`, but only when installing from source (not wheels).

Reproducer:
`pip install  --force-reinstall --ignore-installed --no-binary :all: ruamel.yaml.clib`

```
Snip...
distutils.errors.DistutilsClassError: command class <class 'setuptools.command.egg_info.egg_info'> must subclass Command
```

Workaround
`export SETUPTOOLS_USE_DISTUTILS="stdlib"`
`pip install  --force-reinstall --ignore-installed --no-binary :all: ruamel.yaml.clib`

```
Collecting ruamel.yaml.clib
  Downloading https://files.pythonhosted.org/packages/92/28/612085de3fae9f82d62d80255d9f4cf05b1b341db1e180adcf28c1bf748d/ruamel.yaml.clib-0.2.0.tar.gz (178kB)
     |████████████████████████████████| 184kB 739kB/s 
Installing collected packages: ruamel.yaml.clib
    Running setup.py install for ruamel.yaml.clib ... done
Successfully installed ruamel.yaml.clib-0.2.0
```


@asmacdo I can confirm that the workaround you describe with `export SETUPTOOLS_USE_DISTUTILS="stdlib"` seems to resolve the error I am seeing as well.

Note there are seemingly two errors reported here. One relating to `Command` subclass, which since it's in the title and OP, I'll continue to address here. The other, relating to import of pip, was reported in #2365, so I'll use that ticket to track that issue.
Hi @jay-karimi . Sorry for the trouble you experienced and thanks for filing a report. @asmacdo Thanks for the reproducer. That helps a lot. I'm able to replicate the failure with this command:

```
pip-run -q setuptools==50 -- -m pip-run --no-binary :all: --no-deps ruamel.yaml.clib
```
Same when executing against the source repo:

```
$ pip-run -q setuptools==50 -- -m pip-run hg+http://hg.code.sf.net/p/ruamel-yaml-clib/code@0.2.0
```
I traced the issue to its source:

```
ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror -m pdb setup.py egg_info
> /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(5)<module>()
-> from __future__ import print_function, absolute_import, division, unicode_literals
(Pdb) c
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(30)clear_distutils()
-> warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1703, in main
    pdb._runscript(mainpyfile)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py", line 1572, in _runscript
    self.run(statement)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/bdb.py", line 587, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 5, in <module>
    from __future__ import print_function, absolute_import, division, unicode_literals
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 885, in main
    nsp.check()
  File "/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py", line 451, in check
    from pip.exceptions import InstallationError
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 74, in find_spec
    return method()
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 94, in spec_for_pip
    clear_distutils()
  File "/var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py", line 30, in clear_distutils
    warnings.warn("Setuptools is replacing distutils.")
UserWarning: Setuptools is replacing distutils.
(Pdb) w
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py(1703)main()
-> pdb._runscript(mainpyfile)
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pdb.py(1572)_runscript()
-> self.run(statement)
  /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/bdb.py(587)run()
-> exec(cmd, globals, locals)
  <string>(1)<module>()
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(5)<module>()
-> from __future__ import print_function, absolute_import, division, unicode_literals
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(885)main()
-> nsp.check()
  /Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(451)check()
-> from pip.exceptions import InstallationError
  <frozen importlib._bootstrap>(991)_find_and_load()
  <frozen importlib._bootstrap>(971)_find_and_load_unlocked()
  <frozen importlib._bootstrap>(914)_find_spec()
  /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(74)find_spec()
-> return method()
  /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(94)spec_for_pip()
-> clear_distutils()
> /var/folders/qs/5jptvz2x7_gblx4kc3qj005800n8zm/T/pip-run-ixe5il38/_distutils_hack/__init__.py(30)clear_distutils()
-> warnings.warn("Setuptools is replacing distutils.")
```

Notice `/Users/jaraco/draft/ruamel-yaml-clib-code/setup.py(451)check()` there in the call stack when the warning is raised? It seems the setup.py for that project is importing pip (and merely to get an `InstallationError` exception class).

By importing pip, the project is triggering the same behavior reported in #2351. In this case, however, before the `KeyError` is triggered, the distutils hack is disabled, then distutils is imported from the stdlib, then setuptools is imported, and `distutils.cmd.Command` now has two manifestations.

Replacing the use of `import pip` fixes the error:

```
ruamel-yaml-clib-code default $ hg diff
diff -r 52bf0eb35c43 setup.py
--- a/setup.py  Fri Sep 27 07:42:41 2019 +0200
+++ b/setup.py  Tue Sep 01 18:07:28 2020 -0400
@@ -447,10 +447,7 @@
             sys.exit(1)
 
     def check(self):
-        try:
-            from pip.exceptions import InstallationError
-        except ImportError:
-            return
+        InstallationError = Exception
         # arg is either develop (pip install -e) or install
         if self.command not in ['install', 'develop']:
             return

ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror setup.py egg_info
sys.argv ['setup.py', 'egg_info']
test compiling test_ruamel_yaml
running egg_info
writing ruamel.yaml.clib.egg-info/PKG-INFO
writing dependency_links to ruamel.yaml.clib.egg-info/dependency_links.txt
writing namespace_packages to ruamel.yaml.clib.egg-info/namespace_packages.txt
writing top-level names to ruamel.yaml.clib.egg-info/top_level.txt
reading manifest file 'ruamel.yaml.clib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'ruamel.yaml.clib.egg-info/SOURCES.txt'
```

My recommendation is for that project (and any other projects using that technique) avoid importing pip.
@jaraco should we open a pip ticket to discuss a warning if pip is imported in setup.py (is that even possible)?
I apply the same fix for Windowns build and it worked too. Here is Window statement:

```
set SETUPTOOLS_USE_DISTUTILS="stdlib"
pip install ...
```
```
> I traced the issue to its source:
> 
> ```
> ruamel-yaml-clib-code default $ pip-run -q setuptools==50 -- -Werror -m pdb setup.py egg_info
<snip>
> By importing pip, the project is triggering the same behavior reported in #2351. In this case, however, before the `KeyError` is triggered, the distutils hack is disabled, then distutils is imported from the stdlib, then setuptools is imported, and `distutils.cmd.Command` now has two manifestations.
> 
> Replacing the use of `import pip` fixes the error:
> 
<snip> 
> My recommendation is for that project (and any other projects using that technique) avoid importing pip.
```

Thanks for the analysis and suggested fix.
I am in the process of implementing this, this is a common setup.py for all projects ruamel distributes, and especially generating the wheels for C extensions (such as ruamel.yaml.clib) will take some effort. I will however prioritize the open source pages (~25 or so from the 150+ that need updating).
> should we open a pip ticket to discuss a warning if pip is imported in setup.py (is that even possible)?

It is likely possible. Given that for the distutils hack, [setuptools special-cases pip](https://github.com/pypa/setuptools/blob/4c29a5122247f0461a9e44ff8484159c563935c4/_distutils_hack/__init__.py#L90-L96), it may make sense to be even more sophisticated there and not disable the override unless pip is being executed.

Resolving patch:
diff --git a/_distutils_hack/__init__.py b/_distutils_hack/__init__.py
--- a/_distutils_hack/__init__.py
+++ b/_distutils_hack/__init__.py
@@ -92,9 +92,22 @@ def spec_for_pip(self):
         Ensure stdlib distutils when running under pip.
         See pypa/pip#8761 for rationale.
         """
+        if self.pip_imported_during_build():
+            return
         clear_distutils()
         self.spec_for_distutils = lambda: None
 
+    @staticmethod
+    def pip_imported_during_build():
+        """
+        Detect if pip is being imported in a build script. Ref #2355.
+        """
+        import traceback
+        return any(
+            frame.f_globals['__file__'].endswith('setup.py')
+            for frame, line in traceback.walk_stack(None)
+        )
+
 
 DISTUTILS_FINDER = DistutilsMetaFinder()
 


Test patch: