Repository: spack/spack

Issue:
mpl_toolkits.basemap fails to import
I'm having trouble importing mpl_toolkits.basemap. I know this had previously worked, so I don't know where to point the finger. Here's what I see:

```
[lee218@rzwiz2:spack.basemap]$ /nfs/tmp2/lee218/delete/spack.basemap/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-2.7.13-gcx7wpxjtt7ujlpbahezikqfz4xn2ul6/bin/python -c 'import mpl_toolkits; print(mpl_toolkits.__path__); import mpl_toolkits.basemap'
['/nfs/tmp2/lee218/delete/spack.basemap/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-2.7.13-gcx7wpxjtt7ujlpbahezikqfz4xn2ul6/lib/python2.7/site-packages/matplotlib-2.0.0-py2.7-linux-x86_64.egg/mpl_toolkits']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named basemap

[lee218@rzwiz2:spack.basemap]$ /nfs/tmp2/lee218/delete/spack.basemap/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-3.6.0-ndtiujzzf7p4m5t4k6xergu3f2u674sk/bin/python3 -c 'import mpl_toolkits; print(mpl_toolkits.__path__); import mpl_toolkits.basemap'
['/nfs/tmp2/lee218/delete/spack.basemap/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-3.6.0-ndtiujzzf7p4m5t4k6xergu3f2u674sk/lib/python3.6/site-packages/matplotlib-2.0.0-py3.6-linux-x86_64.egg/mpl_toolkits']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'mpl_toolkits.basemap'
```

With a previous installation I see this:

```
[lee218@rzwiz2:spack.basemap]$ python -c 'import mpl_toolkits; print(mpl_toolkits.__path__); import mpl_toolkits.basemap' ['/collab/usr/gapps/python/build/spack-toss3.1/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-2.7.13-gv7myy57t2h6wp44f42b2llku7kiva7z/lib/python2.7/site-packages/matplotlib-1.5.3-py2.7-linux-x86_64.egg/mpl_toolkits', '/collab/usr/gapps/python/build/spack-toss3.1/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-2.7.13-gv7myy57t2h6wp44f42b2llku7kiva7z/lib/python2.7/site-packages/mpl_toolkits']
```

Notice that the __path__ in the latter case includes the mpl_toolkits directory where basemap exists. I'm not sure why the old Spack python build was able to find both the matplotlib and basemap mpl_toolkits directories, while the new Spack python build only finds the maplotlib one. Any thoughts @tgamblin @citibeth @adamjstewart @mathstuf?

I found one hack that works around this, which is to add `__path__.append('/nfs/tmp2/lee218/delete/spack.basemap/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/py-basemap-1.0.7-cs5yvwpwtosqpl3t7rhadlccinbmsqvr/lib/python3.6/site-packages/mpl_toolkits')` to .../matplotlib.../mpl_toolkits/__init__.py.


Discussion:
See #1964. This may have introduced the bug. It sounds like it was meant to address the bug.
Ah yes, this issue rears its ugly head again. @adamjstewart is right, see #1964; and while you're at it, you can also look at #1948 and https://github.com/matplotlib/matplotlib/issues/5327 and https://www.python.org/dev/peps/pep-0420/ .  See in particular the solution I use in #1964 for this issue, which explains some of what you observed:

> This PR solves the problem by symlinking all of the mpl_tools directory from matplotlib into basemap. It's an un-holy mess, and wouldn't scale if there were additional packages that also implicitly "merge" into mpl_tools (there are none that I know of). But it solves the problem at hand, it doesn't break the Spack paridigm, and is better than any other solution I could think of; for example, to create a Spack package that installs Matplotlib and Basemap simultaneously into one Spack prefix.

Based on that, here's where I would look:

1. I see that you are running Python 2.7 and 3.6 in the test above.  In #1964, I only did the symlinking for `python@3.5.2:`, because it had worked for me previously without symlinking.  I didn't understand why it had *ever* worked for me.  But I put in that check to be conservative.  Maybe something changed and the latest Python 2.7 now also needs the patch?  I don't know...  Same thing (but different) for 3.6... maybe Python 3.6 did something that my patch isn't needed?  Anyway, I would try turning the patch on and off with your different favored Python versions, and see if it makes any difference.  (You can prototype that by symlinking the files yourself, you don't have to go through an entire install to try it out).

2. When I added the patch, it was part of a single monolithic `install()` method.  A lot has changed since then: we now have `PythonPackage` and `@run_after('install') def post_install_patch()`.  I would make sure that the patch is actually being run at all... look for the symlinks in the `basemap` install tree, over to `matplotlib`.  Or add debugging printout.

3. Maybe there's something strange about your filesystem where the symlinks are there, but they don't work from the account/setup you're trying to access them from.  ???

4. Try installing with `python@3.5.2` and see if it "just works."  That's the version I've used.

5. Are you using `spack activate`, or loading modules?  If `spack activate`, try loading modules instead and see if it works then.  I could imagine the following scenario that could result in #1964 not taking effect
```
spack install py-matplotlib
spack activate py-matplotlib
spack install py-basemap
spack activate py-basemap
```
In that case, `py-matplotlib` would have been symlinked into `python` *before* `py-basemap` built; and `py-basemap` would NOT be symlinked in there.  In this case, the solution would be to install both packages before activating --- and to adtivate only `py-basemap`:
```
spack install py-matplotlib
spack install py-basemap
spack activate py-basemap
```



@adamjstewart I don't think that introduced the bug. I see the same problem with version 2.7.13, while #1964 only patches version 3.5.2 and greater.
Try changing the if statement in 1964 to apply the patch to other versions
uf python as well.  It should work just as well.
On Apr 13, 2017 10:49 AM, "Gregory Lee" <notifications@github.com> wrote:

> @adamjstewart <https://github.com/adamjstewart> I don't think that
> introduced the bug. I see the same problem with version 2.7.13, while
> #1964 <https://github.com/LLNL/spack/pull/1964> only patches version
> 3.5.2 and greater.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/LLNL/spack/issues/3813#issuecomment-293917430>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AB1cd3Ky5gYc7zlEqBPndYbP3p1fe_h5ks5rvjXWgaJpZM4M75XO>
> .
>

@citibeth #1964 was not the cause of the problem. Note that in my tests above that Python 2.7.13 failed in the same way. ~~However, I did try your suggestion of this:~~

```
spack install py-matplotlib
spack activate py-matplotlib
spack install py-basemap
spack activate py-basemap
```

~~And that ran OK! Here's the output:~~

```
[lee218@rzwiz2:spack.basemap]$ /nfs/tmp2/lee218/delete/spack.basemap/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-3.6.0-ndtiujzzf7p4m5t4k6xergu3f2u674sk/bin/python3 -c 'import mpl_toolkits; print(mpl_toolkits.__path__); import mpl_toolkits.basemap'
['/nfs/tmp2/lee218/delete/spack.basemap/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/python-3.6.0-ndtiujzzf7p4m5t4k6xergu3f2u674sk/lib/python3.6/site-packages/matplotlib-2.0.0-py3.6-linux-x86_64.egg/mpl_toolkits', '/nfs/tmp2/lee218/delete/spack.basemap/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/py-basemap-1.0.7-cs5yvwpwtosqpl3t7rhadlccinbmsqvr/lib/python3.6/site-packages/mpl_toolkits']
```

~~Note that both mpl_toolkits directories are now reported in `mpl_toolkits.__path__`. I will do some more investigation to see if I can see what difference this makes.~~
Ugh, sorry, my previous comment is inaccurate as I had my workaround (modifying matplotlibs `mpl_toolkits.__init__.py` to include basemap's `__path__`) in place. Activating maplotlib before installing basemap doesn't help. More testing ensues...

Please note, though, that #1964 does not appear to be the root of the problem. Also, the linking done in #1964 still occurs when activating matplotlib before installing basemap.
Can you try loading modules instead of activating?

Can you post a 'ls -lR' of the matplotlib and basemap install dirs?
On Apr 13, 2017 3:50 PM, "Gregory Lee" <notifications@github.com> wrote:

> Ugh, sorry, my previous comment is inaccurate as I had my workaround
> (modifying matplotlibs mpl_toolkits.__init__.py to include basemap's
> __path__) in place. Activating maplotlib before installing basemap
> doesn't help. More testing ensues...
>
> Please note, though, that #1964 <https://github.com/LLNL/spack/pull/1964>
> does not appear to be the root of the problem. Also, the linking done in
> #1964 <https://github.com/LLNL/spack/pull/1964> still occurs when
> activating matplotlib before installing basemap.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/LLNL/spack/issues/3813#issuecomment-294004182>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AB1cd3PNkKbaX-vjRqwm68RT7YPO_XPOks5rvnxpgaJpZM4M75XO>
> .
>

@citibeth if I do not activate py-matplotlib and do not activate py-basemap, but I do load the py-basemap module, then it runs OK. This is not surprising, though, since in this case it only finds py-basemap's mpl_toolkits directory. The problem when things are activated is that Python first finds matplotlib's mpl_toolkits directory and then stops looking.
So you have two solutions here:

1. Load modules, don't activate.  We know this works.  Activation is
entirely optional.

2. Try activating just basemap and not matplotlib.  I think this will work.

Or maybe figure out a better fix to speck activate.
On Apr 13, 2017 4:59 PM, "Gregory Lee" <notifications@github.com> wrote:

@citibeth <https://github.com/citibeth> if I do not activate py-matplotlib
and do not activate py-basemap, but I do load the py-basemap module, then
it runs OK. This is not surprising, though, since in this case it only
finds py-basemap's mpl_toolkits directory. The problem when things are
activated is that Python first finds matplotlib's mpl_toolkits directory
and then stops looking.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/LLNL/spack/issues/3813#issuecomment-294020083>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cdy-5D1nOSH0FZ3eBrMGNMLwfJo4qks5rvoyqgaJpZM4M75XO>
.

FWIW, it appears that it is something in the Spack build of Python and basemap that is causing this. I just installed Python 3.6.0 + numpy + maplotlib + basemap from source outside of Spack and that runs OK. When I run my simple test, I see both mpl_toolkits dir in the packages `__path__`:

```
[lee218@rzwiz2:dummy]$ /nfs/tmp2/lee218/prefix/python-3.6.0/bin/python3 -c 'import mpl_toolkits; print(mpl_toolkits.__path__); import mpl_toolkits.basemap'
['/nfs/tmp2/lee218/prefix/python-3.6.0/lib/python3.6/site-packages/mpl_toolkits', '/nfs/tmp2/lee218/prefix/python-3.6.0/lib/python3.6/site-packages/matplotlib-2.0.0-py3.6-linux-x86_64.egg/mpl_toolkits']
```
Upstream didn't follow PEP420.  Anything else is a hack.
On Apr 13, 2017 7:11 PM, "Gregory Lee" <notifications@github.com> wrote:

> FWIW, it appears that it is something in the Spack build of Python and
> basemap that is causing this. I just installed Python 3.6.0 + numpy +
> maplotlib + basemap from source outside of Spack and that runs OK. When I
> run my simple test, I see both mpl_toolkits dir in the packages __path__:
>
> [lee218@rzwiz2:dummy]$ /nfs/tmp2/lee218/prefix/python-3.6.0/bin/python3 -c 'import mpl_toolkits; print(mpl_toolkits.__path__); import mpl_toolkits.basemap'
> ['/nfs/tmp2/lee218/prefix/python-3.6.0/lib/python3.6/site-packages/mpl_toolkits', '/nfs/tmp2/lee218/prefix/python-3.6.0/lib/python3.6/site-packages/matplotlib-2.0.0-py3.6-linux-x86_64.egg/mpl_toolkits']
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/LLNL/spack/issues/3813#issuecomment-294044659>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AB1cd2_8gFLpByeQXSj3Pv-eJ9CAi1Pkks5rvquHgaJpZM4M75XO>
> .
>


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-basemap/package.py b/var/spack/repos/builtin/packages/py-basemap/package.py
--- a/var/spack/repos/builtin/packages/py-basemap/package.py
+++ b/var/spack/repos/builtin/packages/py-basemap/package.py
@@ -23,7 +23,6 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
 from spack import *
-import os
 
 
 class PyBasemap(PythonPackage):
@@ -35,6 +34,10 @@ class PyBasemap(PythonPackage):
 
     version('1.0.7', '48c0557ced9e2c6e440b28b3caff2de8')
 
+    # Per Github issue #3813, setuptools is required at runtime in order
+    # to make mpl_toolkits a namespace package that can span multiple
+    # directories (i.e., matplotlib and basemap)
+    depends_on('py-setuptools', type=('build', 'run'))
     depends_on('py-numpy', type=('build', 'run'))
     depends_on('py-matplotlib', type=('build', 'run'))
     depends_on('pil', type=('build', 'run'))
@@ -42,79 +45,3 @@ class PyBasemap(PythonPackage):
 
     def setup_environment(self, spack_env, run_env):
         spack_env.set('GEOS_DIR', self.spec['geos'].prefix)
-
-    @run_after('install')
-    def post_install_patch(self):
-        spec = self.spec
-        # We are not sure if this fix is needed before Python 3.5.2.
-        # If it is needed, this test should be changed.
-        # See: https://github.com/LLNL/spack/pull/1964
-        if spec['python'].version >= Version('3.5.2'):
-            # Use symlinks to join the two mpl_toolkits/ directories into
-            # one, inside of basemap.  This is because Basemap tries to
-            # "add to" an existing package in Matplotlib, which is only
-            # legal Python for "Implicit Namespace Packages":
-            #     https://www.python.org/dev/peps/pep-0420/
-            #     https://github.com/Homebrew/homebrew-python/issues/112
-            # In practice, Python will see only the basemap version of
-            # mpl_toolkits
-            path_m = find_package_dir(
-                spec['py-matplotlib'].prefix, 'mpl_toolkits')
-            path_b = find_package_dir(spec.prefix, 'mpl_toolkits')
-            link_dir(path_m, path_b)
-
-
-def find_package_dir(spack_package_root, name):
-
-    """Finds directory with a specific name, somewhere inside a Spack
-    package.
-
-    spack_package_root:
-        Root directory to start searching
-    oldname:
-        Original name of package (not fully qualified, just the leaf)
-    newname:
-        What to rename it to
-
-    """
-    for root, dirs, files in os.walk(spack_package_root):
-        path = os.path.join(root, name)
-
-        # Make sure it's a directory
-        if not os.path.isdir(path):
-            continue
-
-        # Make sure it's really a package
-        if not os.path.exists(os.path.join(path, '__init__.py')):
-            continue
-
-        return path
-
-    return None
-
-
-def link_dir(src_root, dest_root, link=os.symlink):
-    """Link all files in src_root into directory dest_root"""
-
-    for src_path, dirnames, filenames in os.walk(src_root):
-        if not filenames:
-            continue        # avoid explicitly making empty dirs
-
-        # Avoid internal Python stuff
-        src_leaf = os.path.split(src_path)[1]
-        if src_leaf.startswith('__'):
-            continue
-
-        # Make sure the destination directory exists
-        dest_path = os.path.join(dest_root, src_path[len(src_root) + 1:])
-        try:
-            os.makedirs(dest_path)
-        except:
-            pass
-
-        # Link all files from src to dest directory
-        for fname in filenames:
-            src = os.path.join(src_path, fname)
-            dst = os.path.join(dest_path, fname)
-            if not os.path.exists(dst):
-                link(src, dst)
diff --git a/var/spack/repos/builtin/packages/py-matplotlib/package.py b/var/spack/repos/builtin/packages/py-matplotlib/package.py
--- a/var/spack/repos/builtin/packages/py-matplotlib/package.py
+++ b/var/spack/repos/builtin/packages/py-matplotlib/package.py
@@ -59,7 +59,10 @@ class PyMatplotlib(PythonPackage):
     extends('python', ignore=r'bin/nosetests.*$|bin/pbr$')
 
     # ------ Required dependencies
-    depends_on('py-setuptools', type='build')
+    # Per Github issue #3813, setuptools is required at runtime in order
+    # to make mpl_toolkits a namespace package that can span multiple
+    # directories (i.e., matplotlib and basemap)
+    depends_on('py-setuptools', type=('build', 'run'))
 
     depends_on('libpng@1.2:')
     depends_on('freetype@2.3:')


Test patch: