Repository: spack/spack

Issue:
Python 3 does not install pip
Python's package contains the following two lines:

``` bash
if spec.satisfies('@3:'):
    config_args.append('--without-ensurepip')
```

Which dates back to c65fd3a28968f549eed361d85f715caa4f8f6436. I was able to install python with pip after commenting out this line. Is this flag necessary, or could pip be a variant? Pip may be another package manager, but it's not superfluous.



Discussion:
To be safe, I would make `+pip` be a variant.

I found myself wanting a `pip3` related to my `spack` installed `python@3.5.2` as well.  Assuming the module has been loaded (aka `python3` refers to the `spack` python), I think the above behavior is safest -- by default assume no `pip3` installed.

I like the `+pip` variant idea, the modulefile already consistently makes `pip3` point to the right one :)

If users did not receive `pip`, they can do
```
python3 -m ensurepip
```
after the fact.

There should be cause for alarm, though.  Maybe just create extra clear documentation on mixing the two, putting the burden on the user to be extra careful about the order in which things are executed (read: always try and `spack install py-PKG` before falling back to `pip3 install PKG`).

### Scenario 1: Good User

This user knows full-well that `spack` is going to give them the best `numpy` installation around, and they have loaded into their environment `python@3.5.2` (as an example) and `py-numpy`
```
pip3 install numpy
Requirement already satisfied (use --upgrade to upgrade): numpy in /home/sven/spack/opt/spack/linux-fedora23-x86_64/gcc-5.3.1/py-numpy-1.11.2-jo7wrrrlpqth5rxpxynajnee44c26ua2/lib/python3.5/site-packages/numpy-1.11.2-py3.5-linux-x86_64.egg
```

### Scenario 2: Unconscious User

They only did say `spack load python@3.5.2`, and now they do
```
> pip3 install numpy
Collecting numpy
  Downloading numpy-1.11.2-cp35-cp35m-manylinux1_x86_64.whl (15.6MB)
    58% |██████████████████▊             | 9.1MB 18.3MB/s eta 0:00:01^C
Operation cancelled by user
```

EEEK!  I can only imagine this could cause some trauma for users if there are conflicts.

### Discussion

What are other people's thoughts?  I personally am wanting to find a way to make this process 'safe' in terms of finding a way to maybe at least warn of the unconscious scenario.  I'd be interested in helping implement this, but don't really know if (1) it is possible and (2) how to go about it cleanly.

Given the already special treatment of `python` (e.g. specifically packaged `py-*`), would it be a good idea to at least add a specific docs page discussing the intricacies of `spack` and `python`?

Aside from `pip`, I am aware of other nuances that do not seem to be documented such as if a spec asks for `python`, it will default to `2.7`.  To be clear, I agree with this choice.  But it only came to my attention after-the-fact.

There are a great number of packages available through `pip` that (in my opinion) shouldn't ever be packaged into `spack`.  ~E.g. `sphinx` or `ipdb` or something~.  But at the same time, having a full-blown installation of `python` via `spack` that in some senses prohibits the user from being able to install a debugger or something seems to render this python less useful than it otherwise could be.

Any thoughts on what the "right" solution here is, and anything I can help do to make that come true?

**Edit**: ok the two examples I listed are already packaged in `spack`.  I guess this is part of what I'm getting at, are these just added when somebody decides "hey I want this package", or are there specific reasons as to why `py-X` is already present in `spack`?

**Edit 2**: a better example is why the existing defaults of `python` will **not** install the `tk` variant (~why?~ `spack` is targeting clusters, this makes sense), but then if I want to install `py-matplotlib` I must either re-install python to incorporate `tk` or install `qt` as my graphics backend.

Just trying to get a discussion going about the nuances to better document it if desired.
@sknigh I looked at it a bit... I have no idea why PIP is disabled for Python3.  I would either (a) just remove it, or (b) make it an option that defaults to `+pip`.  I don't find PIP particularly useful in the Spack context, but its presence shouldn't be harmful.

> If users did not receive pip, they can do

This is problematic because ideally, the same hash should always equal the same install tree.  If you start writing into install trees after they get installed, this breaks that core assumption about the way Spack works; and could lead to other "mysterious" problems.  Unfortunately, sometimes this happens anyway; but I don't think it should be encouraged or condoned.

If you want to use PIP, it should AT LEAST be used to install packages somewhere OTHER than the Spack-built Python tree.  It's not hard to configure PIP to work this way.

> (read: always try and spack install py-PKG before falling back to pip3 install PKG).

Or how about... create `py-PKG/package.py` if it's not already there.  They are very easy to create from the PIP website.  @adamjstewart should we try to make this more automagic?  I think that's a far better solution than trying to mix PIP and Spack.

> They only did say spack load python@3.5.2, and now they do

There are two ways to use Python in Spack: either load the modules you need via `spack load` (or equivalent); or "activate" the modules you want into your Spack-built Python distro (thereby violating the above assumption, but in a controlled way that IS supported by Spack).  In either case, the user shouldn't be blithely running PIP.

> are these just added when somebody decides "hey I want this package"

Exactly.

> why the existing defaults of `python` will **not** install the `tk` variant (~why?~ `spack` is targeting clusters, this makes sense), but then if I want to install `py-matplotlib` I must either re-install python to incorporate `tk` or install `qt` as my graphics backend.

???  In my experience, I was able to install (and use successfully) the Tk-based GUI on top of the plain Python3.  Have you tried this and seen it not work?

I think you're right, a section of the manual describing Python usage in more detail would be relevant.  The devil is in figuring out what to cover in an organized way.  Do you have specific ideas?  (If so, please start another Isssue thread; it will be tagged `documentation` and whatever outline / text you put in there will eventually make it into the manual).  Or just submit a documentation PR...




@sknigh Just fyi, there are two issues you may want to read. #1249 is raising the general issue of what we should do when we enter into the territory of other package managers. #2281 is instead more focused on PyPI. They are both in a discussion stage right now.
> Or how about... create py-PKG/package.py if it's not already there. They are very easy to create from the PIP website. @adamjstewart should we try to make this more automagic? I think that's a far better solution than trying to mix PIP and Spack.

That's the plan. I think there are quite a few open discussions about this. The way I envision it, you should be able to run:
```
spack create <pypi-url>
```
And Spack will create the package, add the correct homepage and URL based on #2281, and subclass a new PythonPackage class. At that point, the only thing we aren't currently doing would be to automatically add the dependencies and version constraints. I would have to think more about how hard that would be.
@citibeth thank you for the in-depth response.  I will post a couple of separate issues in free time over the next week explaining the `tk` issues, living docs collab issue, etc, and keep this thread focused on the `pip` front.

### Python + pip && Defaults

I would like to submit a PR that standardizes this behavior, adding a `+pip` variant and having it default to **ON**.

1. In the docs page we will discourage adding `pip` in after the fact (using `pythonX -m ensurepip`).
2. Question: given that currently this only affects `python3`, will adding in a variant like this that adds `+pip` cause problems for existing python**2** installations? (where once there was no variant, now there is)

### `pip` via `spack` and Where to Install

It was discussed that having `pip` install to a different location would be ideal.

1. Are you saying this should not install **anything** underneath the `spack` installed python directory?
    - If so, are there specific locations that are safe?
    - Is there going to be a discrepancy if the user does `load` vs `activate`?
        - I've already enlisted @adamjstewart to help me with the docs on this, my understanding is `activate` will be **heavily** encouraged.
2. The discussion seemed to have concluded with "Ideally, if you were about to `pip install X` then we would greatly appeciate you instead quickly creating a `spack` `py-X` for us (pretty please)".
    - If this is the ultimate goal, should we go further and instead default `pip` to **OFF**?  This model basically says "we can't guarantee sanity unless you let `spack` manage this. best success from `spack install py-X`, use `pip` at extreme risk" or something?
    
> I would like to submit a PR that standardizes this behavior, adding a `+pip` variant and having it default to **ON**.

Sounds like a worth cause for a PR.

> 1. In the docs page we will discourage adding `pip` in after the fact (using `pythonX -m ensurepip`).

What does that command do?

> 2. Question: given that currently this only affects `python3`, will adding in a variant like this that adds `+pip` cause problems for existing python**2** installations? (where once there was no variant, now there is)

I don't know, but probably not.  It would be nice to know why the PIP stuff was only added for `python3`.  The answer might be as simple as, the person who added it used Python3 and wanted to be conservative.  Would you be able to go through `git blame`, figure out who it is, and ask that person?


> 1. Are you saying this should not install **anything** underneath the `spack` installed python directory?

Yes

> Anywhere not Spack-controlled.  For example, in your home directory.  Or in some shared directory.  See:
>    - If so, are there specific locations that are safe?

http://stackoverflow.com/questions/2915471/install-a-python-package-into-a-different-directory-using-pip
https://www.dabapps.com/blog/introduction-to-pip-and-virtualenv-python/
https://www.zoomeranalytics.com/blog/pip-install-t
http://conda.pydata.org/docs/_downloads/conda-pip-virtualenv-translator.html
https://docs.python.org/3/install/index.html#alternate-installation-the-user-scheme

> - Is there going to be a discrepancy if the user does `load` vs `activate`?

Things installed with `pip` would be neither `load`ed nor `activate`d.  Depending on how they are installed (as you can see, there are many options), they would be picked up automatically in the user's home directory, or found in a virtualenv, or something.

> - I've already enlisted @adamjstewart to help me with the docs on this, my understanding is `activate` will be **heavily** encouraged.

People like `activate`, but we must keep in mind that it breaks Spack in a fundamental way.  With Spack in C-land, you can you can build two versions of `mypackage@1.0` that depend on two different versions of `openmpi`; and you can run those two simultaneously.  You can do something similar in Python land using `spack load`: for example, build two versions of `py-mypackage@1.0` that depend on two different versions of `py-numpy`; and run them simultaneously.  You cannot do that if you use `spack activate`.

Many use cases don't care about this, and `spack activate` works fine there.  But for some, it could be a problem.  So we must document and support `spack load` just as well.

Of course `spack load` has its own problems: namely, that it relies on setting `$PYTHONPATH` to find things.  And `$PYTHONPATH` is the evil twin to `$LD_LIBRARY_PATH`; the two are both evil for all the same reasons.  `RPATH` is the antidote to `$LD_LIBRARY_PATH`, so maybe we could have something similar in Python land?  See #1927 ("RPATH in Python Code") for a possible way to fix this.  This idea would take some development and testing to be useful.  (OT... it looks like #2506 might have fixed the RPATH problem in compiled Python modules).

The other thing to keep in mind with Python packages is that dependencies are not as strict as with compiled code.  To a large extent, package versions can be mix-n-matched at runtime.  For example... if I installed `py-a^numpy@1.8`, I can later decide to run `py-a^numpy@1.7` and it will PROBABLY work (and sometimes not work).  Conda makes use of this "looseness" of versioning, and Spack does not.  With Spack, if you built with one version of `numpy`, that's the only version Spack will let you run with.  I would say that Spack's approach is certainly safer, even if it disallows some combinations that really are just fine.




> 2. The discussion seemed to have concluded with "Ideally, if you were about to `pip install X` then we would greatly appeciate you instead quickly creating a `spack` `py-X` for us (pretty please)".

I would say so.

> - If this is the ultimate goal, should we go further and instead default `pip` to **OFF**?

Because PIP is part of Python; and as shown above, it can be used for many purposes other than messing up Spack.  You just have to not install into "system" locations.  This is actually the way PIP is normally used by normal users, who do not have write access to the System-installed Python locations.  Really, you should never be writing stuff into your may Python, no matter how it was installed.

Hey Again!

Sorry for the delay, I've been slowly writing up py docs / "best practices" for spack && python, part of which directly relates to the pip dilemma.  Rather unexpected deadlines between semesters arose that have negated all free time I originally allocated for writing this...

### Origins of the no-pip-for-python3

Don't really seem to matter too much, but looking at the [blame](https://github.com/LLNL/spack/blame/develop/var/spack/repos/builtin/packages/python/package.py#L139) it seems to have just sort of been inserted with [another fix](https://github.com/LLNL/spack/commit/d85a46380cd0b2b113c8e3de111298abff2dd24f).  @lee218llnl seems to have added it (I think, the name doesn't link on GitHub), but I agree with @citibeth that using this as a variant shouldn't cause issues.  It hasn't for me at least.

### Considerations for pip revisited

I do not find the following to be *entirely* true:

> You just have to not install into "system" locations. This is actually the way PIP is normally used by normal users, who do not have write access to the System-installed Python locations. Really, you should never be writing stuff into your may Python, no matter how it was installed.

For example, `gnome-tweak-tool` using a spack managed `python@2.7.13+tk` sourced in my `~/.bash_profile` will silently crash because the `gi` module has not been installed.  This module is fundamental used by other operating system tasks related to the desktop manager / settings / etc.

This is where system installed locations are the answer, because if there were other users on my computer I would want to make sure they had the `gi` module too.  Or rather, if I'm making a spack installed python available for multiple users, they will **all** need this module.

I don't know how big of a concern this is for spack though, since the target isn't graphical desktops.  AKA it's nice that it works for desktops, but low priority.

This module **is** available in `py-pygobject`, but I am hoping to establish some finely tuned "spack meets pip" guidelines if possible.

### Answer 1

We (spack) demand that users **only** install packages from PyPi **via spack**.  To my understanding, this is the "ideal" scenario.

Seems to nullify any need for `pip`, and instead no variant is created and removing pip from python 2 would happen as well.

### Answer 2

Give people `pip` as a variant, and provide huge and bold-faced warnings about the dangers of mixing package managers (specifically `pip` and `spack`).

1. In the docs, give the `numpy` example above and basically just say "you break it...your fault".
2. Link to the [python docs on installation customization](https://docs.python.org/3/install/index.html#custom-installation).
    - System administrators could add this in some designated `/etc` file that gets sourced on login for all users?
    - End users (e.g. me on Fedora) can put it in a `$HOME` dotfile

Assuming users followed the directions, the installation should be reproducible (nothing from `pip` is getting installed under the `spack` installation directory).  But if they don't, things could get really messy.

### EOF

I'm sorry if this isn't coherent, but I guess I'm just confused about what to write in these python docs.  Pip is core part of python for many users, but I looked all over and couldn't find a truly global way of containing pip installations automatically **without changing pip itself**.  As in without either setting some config variables, or adding flags to the command line call of `pip`, I don't see how to change this location.

With respect to `activate` vs `load`, I think just explaining the differences (with the examples from @citibeth) should be sufficient (but also necessary!) for overall python how-to and can add it :)

I also don't understand why `virtualenv` or `conda` are concerns here.
- `virtualenv` should "just work" ™️ regardless of what is done with `pip`, right?
- `conda` (in my opinion) is fundamentally incompatible with `spack`, they do the whole stack in-house.  As in it's purely an alternative, not a mix-in...
> using this [the +pip option for Python 3] as a variant shouldn't cause issues. It hasn't for me at least.

Unfortunately, this is not true anymore with Python 3.7 :/ `ensurepip` now requires `ctypes` for some reason, see https://bugs.python.org/issue31652, and `ctypes` doesn't compile the way the `python` package is currently set up, which results in this error:

```
==> Executing phase: 'install'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j32' 'install'

1 error found in build log:
     8911      File "/tmp/tmp7y2lkesw/pip-10.0.1-py2.py3-none-any.whl/pip/_internal/index.py", line 25, in <module>
     8912      File "/tmp/tmp7y2lkesw/pip-10.0.1-py2.py3-none-any.whl/pip/_internal/download.py", line 39, in <module>
     8913      File "/tmp/tmp7y2lkesw/pip-10.0.1-py2.py3-none-any.whl/pip/_internal/utils/glibc.py", line 3, in <module>
     8914      File "/cnk/common/tools/spack/var/spack/stage/python-3.7.0-wr7smr6tublir72kp4koxj33zrmrv6fk/Python-3.7.0/Lib/ctypes/__init__.py", line 7, in <module>
     8915        from _ctypes import Union, Structure, Array
     8916    ModuleNotFoundError: No module named '_ctypes'
  >> 8917    make: *** [install] Error 1
```

I've tried adding `libffi` as a dependency, as recommended on the Python bug tracker, but it still fails, so more digging is needed, which I don't have time for right now...
@dlukes: I'm also facing the same issue. If you are using a Spack-installed libffi, you might want to double-check how it was installed.

Here it seems like something went wrong:
```
/usr/bin/install -c -m 644 ffi.h ffitarget.h '/.../spack/opt/spack/linux-rhel7-ppc64le/gcc-7.1.0/libffi-3.2.1-di5ey4zmp6vaes6ew5iensh7ynp4ym4y/lib/libffi-3.2.1/include'
```
which explains why the header file cannot be found in `/.../spack/opt/spack/linux-rhel7-ppc64le/gcc-7.1.0/libffi-3.2.1-di5ey4zmp6vaes6ew5iensh7ynp4ym4y/include`...

**Edit**: I got it working by

1. adding this Easybuild patch to the current `libffi` package: https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/master/easybuild/easyconfigs/l/libffi/libffi-3.2.1_fix-include-location.patch.

2. adding `libffi` as a dependency to Python and hacking around to get a working configure:
```
        # setup.py needs to be able to read the CPPFLAGS and LDFLAGS
        # as it scans for the library and headers to build
        dep_pfxs = [dspec.prefix for dspec in spec.dependencies('link')]
        config_args = [
            '--with-threads',
            'CPPFLAGS=-I{0}'.format(' -I'.join(dp.include for dp in dep_pfxs)),
            'LDFLAGS=-L{0} {1}'.format(' -L'.join(dp.lib for dp in dep_pfxs), spec['libffi'].libs.search_flags),
        ]
```
I had to do that because `libffi` uses `lib64` instead of `lib`. The `libs` property should probably be used every times but it currently doesn't work for some packages (e.g. `openssl`, `sqlite` and maybe others).
What's the status of this issue? I maintain Spack's `python` recipe, so I can add a `pip` variant if we want to, but there is already a `py-pip` package that can be installed after the fact and is more up-to-date than whatever random version `python` pulls down. We would also have to make sure those don't conflict. I'm learning towards closing this issue and telling users to use `py-pip` if they want to install `pip`.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-build/package.py b/var/spack/repos/builtin/packages/py-build/package.py
--- a/var/spack/repos/builtin/packages/py-build/package.py
+++ b/var/spack/repos/builtin/packages/py-build/package.py
@@ -16,7 +16,7 @@ class PyBuild(PythonPackage):
 
     variant('virtualenv', default=False, description='Install optional virtualenv dependency')
 
-    depends_on('python@3.6:', type=('build', 'run'))
+    depends_on('python@3.6:+ensurepip', type=('build', 'run'))
     depends_on('py-setuptools', type='build')
     depends_on('py-packaging@19:', type=('build', 'run'))
     depends_on('py-pep517@0.9.1:', type=('build', 'run'))
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
+import glob
 import json
 import os
 import platform
@@ -154,12 +155,13 @@ class Python(AutotoolsPackage):
     variant('nis',      default=False, description='Build nis module')
     variant('zlib',     default=True,  description='Build zlib module')
     variant('bz2',      default=True,  description='Build bz2 module')
-    variant('lzma',     default=True,  description='Build lzma module')
+    variant('lzma',     default=True,  description='Build lzma module', when='@3.3:')
     variant('pyexpat',  default=True,  description='Build pyexpat module')
     variant('ctypes',   default=True,  description='Build ctypes module')
     variant('tkinter',  default=False, description='Build tkinter module')
     variant('uuid',     default=True,  description='Build uuid module')
     variant('tix',      default=False, description='Build Tix module')
+    variant('ensurepip', default=True, description='Build ensurepip module', when='@2.7.9:2,3.4:')
 
     depends_on('pkgconfig@0.9.0:', type='build')
     depends_on('gettext +libxml2', when='+libxml2')
@@ -272,7 +274,7 @@ def determine_variants(cls, exes, version_str):
 
         variants = ''
         for module in ['readline', 'sqlite3', 'dbm', 'nis',
-                       'zlib', 'bz2', 'lzma', 'ctypes', 'uuid']:
+                       'zlib', 'bz2', 'ctypes', 'uuid']:
             try:
                 python('-c', 'import ' + module, error=os.devnull)
                 variants += '+' + module
@@ -294,7 +296,23 @@ def determine_variants(cls, exes, version_str):
         except ProcessError:
             variants += '~pyexpat'
 
-        # Some modules changed names in Python 3
+        # Some modules are version-dependent
+        if Version(version_str) >= Version('3.3'):
+            try:
+                python('-c', 'import lzma', error=os.devnull)
+                variants += '+lzma'
+            except ProcessError:
+                variants += '~lzma'
+
+        if Version(version_str) in ver('2.7.9:2,3.4:'):
+            # The ensurepip module is always available, but won't work if built with
+            # --without-ensurepip. A more reliable check to see if the package was built
+            # with --with-ensurepip is to check for the presence of a pip executable.
+            if glob.glob(os.path.join(os.path.dirname(exes[0]), 'pip*')):
+                variants += '+ensurepip'
+            else:
+                variants += '~ensurepip'
+
         if Version(version_str) >= Version('3'):
             try:
                 python('-c', 'import tkinter', error=os.devnull)
@@ -463,8 +481,11 @@ def configure_args(self):
                 raise ValueError(
                     '+ucs4 variant not compatible with Python 3.3 and beyond')
 
-        if spec.satisfies('@3:'):
-            config_args.append('--without-ensurepip')
+        if spec.satisfies('@2.7.9:2,3.4:'):
+            if '+ensurepip' in spec:
+                config_args.append('--with-ensurepip')
+            else:
+                config_args.append('--without-ensurepip')
 
         if '+pic' in spec:
             cflags.append(self.compiler.cc_pic_flag)
@@ -641,6 +662,10 @@ def import_tests(self):
                 else:
                     self.command('-c', 'import Tix')
 
+            # Ensure that ensurepip module works
+            if '+ensurepip' in spec:
+                self.command('-c', 'import ensurepip')
+
     # ========================================================================
     # Set up environment to make install easy for python extensions.
     # ========================================================================


Test patch: