Repository: pypa/setuptools

Issue:
some pip editable installs don't work with PEP 518 build isolation
```shell
> mkdir /tmp/simple_pep518_project-1.0
> printf 'from setuptools import setup; setup(name="simple_pep518_project", version="1.0")\n' >/tmp/simple_pep518_project-1.0/setup.py
> printf '[build-system]\nrequires = ["setuptools", "wheel"]\n' >/tmp/simple_pep518_project-1.0/pyproject.toml
> pip install --prefix /tmp/pyprefix -e /tmp/simple_pep518_project-1.0
Obtaining file:///tmp/simple_pep518_project-1.0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Installing collected packages: simple-pep518-project
  Running setup.py develop for simple-pep518-project
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/tmp/pyprefix:
    running develop
    Checking .pth file support in /tmp/pyprefix/lib/python3.6/site-packages
    /usr/bin/python -E -c pass
    TEST FAILED: /tmp/pyprefix/lib/python3.6/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH
    
    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /tmp/pyprefix/lib/python3.6/site-packages
    
    and your PYTHONPATH environment variable currently contains:
    
        '/tmp/pip-build-env-q4e8bpkm/lib/python3.6/site-packages'
    
    Here are some of your options for correcting the problem:
    
    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files
    
    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)
    
    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:
    
      https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations
    
    
    Please make the appropriate changes for your system and try again.
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/tmp/pyprefix" failed with error code 1 in /tmp/simple_pep518_project-1.0/
> pip install --user -e /tmp/simple_pep518_project-1.0
Obtaining file:///tmp/simple_pep518_project-1.0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Installing collected packages: simple-pep518-project
  Running setup.py develop for simple-pep518-project
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --user not recognized
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=" failed with error code 1 in /tmp/simple_pep518_project-1.0/
```

This is because of pip's build isolation when PEP 518 support is enabled and easy_install's behavior of:
* conflating the python environment used during installation with the target environment, erroneously assuming that the package target directory must be in `PYTHONPATH`.
* disabling support for the `--user` option if the user site is disabled:

```shell
> python easy_install.py -h | grep -- --user                                                                  
  --user                     install in user site-package
> python -s easy_install.py -h | grep -- --user                                                               
> python -s easy_install.py --user setuptools                                                                 
usage: easy_install.py [options] requirement_or_url ...
   or: easy_install.py --help

error: option --user not recognized
```

With easy_install being deprecated, and pip being perfectly happy to install to a target directory not in `PYTHONPATH`, I think:
* the `--user` option should always be available (with maybe a warning if the user site is disabled)
* the error above when installing to a directory not in `PYTHONPATH` should be turned to a warning

For reference, related pip issue: https://github.com/pypa/pip/issues/5317
some pip editable installs don't work with PEP 518 build isolation
```shell
> mkdir /tmp/simple_pep518_project-1.0
> printf 'from setuptools import setup; setup(name="simple_pep518_project", version="1.0")\n' >/tmp/simple_pep518_project-1.0/setup.py
> printf '[build-system]\nrequires = ["setuptools", "wheel"]\n' >/tmp/simple_pep518_project-1.0/pyproject.toml
> pip install --prefix /tmp/pyprefix -e /tmp/simple_pep518_project-1.0
Obtaining file:///tmp/simple_pep518_project-1.0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Installing collected packages: simple-pep518-project
  Running setup.py develop for simple-pep518-project
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/tmp/pyprefix:
    running develop
    Checking .pth file support in /tmp/pyprefix/lib/python3.6/site-packages
    /usr/bin/python -E -c pass
    TEST FAILED: /tmp/pyprefix/lib/python3.6/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH
    
    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /tmp/pyprefix/lib/python3.6/site-packages
    
    and your PYTHONPATH environment variable currently contains:
    
        '/tmp/pip-build-env-q4e8bpkm/lib/python3.6/site-packages'
    
    Here are some of your options for correcting the problem:
    
    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files
    
    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)
    
    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:
    
      https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations
    
    
    Please make the appropriate changes for your system and try again.
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/tmp/pyprefix" failed with error code 1 in /tmp/simple_pep518_project-1.0/
> pip install --user -e /tmp/simple_pep518_project-1.0
Obtaining file:///tmp/simple_pep518_project-1.0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Installing collected packages: simple-pep518-project
  Running setup.py develop for simple-pep518-project
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --user not recognized
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=" failed with error code 1 in /tmp/simple_pep518_project-1.0/
```

This is because of pip's build isolation when PEP 518 support is enabled and easy_install's behavior of:
* conflating the python environment used during installation with the target environment, erroneously assuming that the package target directory must be in `PYTHONPATH`.
* disabling support for the `--user` option if the user site is disabled:

```shell
> python easy_install.py -h | grep -- --user                                                                  
  --user                     install in user site-package
> python -s easy_install.py -h | grep -- --user                                                               
> python -s easy_install.py --user setuptools                                                                 
usage: easy_install.py [options] requirement_or_url ...
   or: easy_install.py --help

error: option --user not recognized
```

With easy_install being deprecated, and pip being perfectly happy to install to a target directory not in `PYTHONPATH`, I think:
* the `--user` option should always be available (with maybe a warning if the user site is disabled)
* the error above when installing to a directory not in `PYTHONPATH` should be turned to a warning

For reference, related pip issue: https://github.com/pypa/pip/issues/5317


Discussion:
I got a similar error message. Does that mean `pip install --user -e .` won't work for projects with `pyproject.toml`?
You mean `pip install --user -e .`?
Oh yes, sorry for the typo.

Benoit Pierre <notifications@github.com> 於 2018年8月2日 週四 01:33 寫道：

> You mean pip install --user -e .?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pypa/setuptools/issues/1405#issuecomment-409657425>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AB2RGTD8cVKieHbdjPMNpCDQcfyCDK5Pks5uMeZogaJpZM4U7Sol>
> .
>

Unfortunately, yes: you'll have to manually install the required build dependencies and use `--no-build-isolation` to work around the issue.
Thanks for the workaround. That works!
This is a problem affecting me big time. I'm trying to install modules with pip into a target for packaging up for AWS lambdas. This breaks a lot of automation (for https://github.com/awslabs/aws-sam-cli)
@pganssle can you confirm a PR doing:


> With easy_install being deprecated, and pip being perfectly happy to install to a target directory not in PYTHONPATH, I think:
>    the --user option should always be available (with maybe a warning if the user site is disabled)
>    the error above when installing to a directory not in PYTHONPATH should be turned to a warning

would be accepted?
@gaborbernat Better to ask @benoit-pierre, he's also a maintainer of `setuptools`, and he's the one who opened the issue. :wink:

I don't have any objection to it, but I also don't understand what consequences there might be. @benoit-pierre were you uncertain about anything in this?
No, I think it's the right move.
Can we make this happen?
I got a similar error message. Does that mean `pip install --user -e .` won't work for projects with `pyproject.toml`?
You mean `pip install --user -e .`?
Oh yes, sorry for the typo.

Benoit Pierre <notifications@github.com> 於 2018年8月2日 週四 01:33 寫道：

> You mean pip install --user -e .?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pypa/setuptools/issues/1405#issuecomment-409657425>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AB2RGTD8cVKieHbdjPMNpCDQcfyCDK5Pks5uMeZogaJpZM4U7Sol>
> .
>

Unfortunately, yes: you'll have to manually install the required build dependencies and use `--no-build-isolation` to work around the issue.
Thanks for the workaround. That works!
This is a problem affecting me big time. I'm trying to install modules with pip into a target for packaging up for AWS lambdas. This breaks a lot of automation (for https://github.com/awslabs/aws-sam-cli)
@pganssle can you confirm a PR doing:


> With easy_install being deprecated, and pip being perfectly happy to install to a target directory not in PYTHONPATH, I think:
>    the --user option should always be available (with maybe a warning if the user site is disabled)
>    the error above when installing to a directory not in PYTHONPATH should be turned to a warning

would be accepted?
@gaborbernat Better to ask @benoit-pierre, he's also a maintainer of `setuptools`, and he's the one who opened the issue. :wink:

I don't have any objection to it, but I also don't understand what consequences there might be. @benoit-pierre were you uncertain about anything in this?
No, I think it's the right move.
Can we make this happen?

Resolving patch:
diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
--- a/setuptools/command/easy_install.py
+++ b/setuptools/command/easy_install.py
@@ -157,19 +157,16 @@ class easy_install(Command):
          "allow building eggs from local checkouts"),
         ('version', None, "print version information and exit"),
         ('no-find-links', None,
-         "Don't load find-links defined in packages being installed")
+         "Don't load find-links defined in packages being installed"),
+        ('user', None, "install in user site-package '%s'" % site.USER_SITE)
     ]
     boolean_options = [
         'zip-ok', 'multi-version', 'exclude-scripts', 'upgrade', 'always-copy',
         'editable',
-        'no-deps', 'local-snapshots-ok', 'version'
+        'no-deps', 'local-snapshots-ok', 'version',
+        'user'
     ]
 
-    if site.ENABLE_USER_SITE:
-        help_msg = "install in user site-package '%s'" % site.USER_SITE
-        user_options.append(('user', None, help_msg))
-        boolean_options.append('user')
-
     negative_opt = {'always-unzip': 'zip-ok'}
     create_index = PackageIndex
 
@@ -273,6 +270,9 @@ def finalize_options(self):
             self.config_vars['userbase'] = self.install_userbase
             self.config_vars['usersite'] = self.install_usersite
 
+        elif self.user:
+            log.warn("WARNING: The user site-packages directory is disabled.")
+
         self._fix_install_dir_for_user_site()
 
         self.expand_basedirs()
@@ -479,8 +479,9 @@ def check_site_dir(self):
                 self.cant_write_to_target()
 
         if not is_site_dir and not self.multi_version:
-            # Can't install non-multi to non-site dir
-            raise DistutilsError(self.no_default_version_msg())
+            # Can't install non-multi to non-site dir with easy_install
+            pythonpath = os.environ.get('PYTHONPATH', '')
+            log.warn(self.__no_default_msg, self.install_dir, pythonpath)
 
         if is_site_dir:
             if self.pth_file is None:
@@ -1311,10 +1312,6 @@ def byte_compile(self, to_compile):
         Please make the appropriate changes for your system and try again.
         """).strip()
 
-    def no_default_version_msg(self):
-        template = self.__no_default_msg
-        return template % (self.install_dir, os.environ.get('PYTHONPATH', ''))
-
     def install_site_py(self):
         """Make sure there's a site.py in the target dir, if needed"""
 


Test patch: