Repository: spack/spack

Issue:
[BUG] No RPATH in Python extensions
See the dialog below for details.  Since Spack does not use hierarchical modules, working around this bug requires that one do `spack load` on an arbitrary and growing set of modules.

```
[me@ankeli spack2]$ spack purge
[me@ankeli spack2]$ spack load py-numpy
[me@ankeli spack2]$ python3 -c 'import numpy'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
```

Uh-oh, what's wrong... let's debug this with `ldd`:

```
[me@ankeli spack2]$ ldd `spack location -i py-numpy`/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffc4ef1f000)
    libopenblas.so.0 => not found
    libm.so.6 => /lib64/libm.so.6 (0x00007f6ccbd1d000)
    libpython3.5m.so.1.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/python-3.5.1-5q5rsrtjld4u6jiicuvtnx52m7tfhegi/lib/libpython3.5m.so.1.0 (0x00007f6ccb7f7000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6ccb5db000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f6ccb219000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6ccc440000)
    libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f6ccb014000)
    libutil.so.1 => /usr/lib64/libutil.so.1 (0x00007f6ccae11000)
```

Same problem, it cannot load `libopenblas.so`.  So let's tell Spack to load it explicitly and try again:

```
[me@ankeli spack2]$ spack load openblas
[me@ankeli spack2]$ ldd `spack location -i py-numpy`/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffc78d2c000)
    libopenblas.so.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/openblas-0.2.17-pw6rmlom7apfsnjtzfttyayzc7nx5e7y/lib/libopenblas.so.0 (0x00007fb61f48e000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fb61f175000)
    libpython3.5m.so.1.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/python-3.5.1-5q5rsrtjld4u6jiicuvtnx52m7tfhegi/lib/libpython3.5m.so.1.0 (0x00007fb61ec4f000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb61ea32000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fb61e671000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb620878000)
    libgfortran.so.3 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/libgfortran.so.3 (0x00007fb61e352000)
    libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fb61e14d000)
    libutil.so.1 => /usr/lib64/libutil.so.1 (0x00007fb61df4a000)
    libquadmath.so.0 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/../lib64/libquadmath.so.0 (0x00007fb61dd0c000)
    libgcc_s.so.1 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/../lib64/libgcc_s.so.1 (0x00007fb61daf6000)
```

OK, now it works with `ldd`.  And try with Python:

```
[me@ankeli spack2]$ python3 -c 'import numpy'
```

Remember... LD_LIBRARY_PATH is EVIL!!!

[BUG] No RPATH in Python extensions
See the dialog below for details.  Since Spack does not use hierarchical modules, working around this bug requires that one do `spack load` on an arbitrary and growing set of modules.

```
[me@ankeli spack2]$ spack purge
[me@ankeli spack2]$ spack load py-numpy
[me@ankeli spack2]$ python3 -c 'import numpy'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/py-numpy-1.11.0-mulodttw5pcyjufva4htsktwty4qd52r/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
```

Uh-oh, what's wrong... let's debug this with `ldd`:

```
[me@ankeli spack2]$ ldd `spack location -i py-numpy`/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffc4ef1f000)
    libopenblas.so.0 => not found
    libm.so.6 => /lib64/libm.so.6 (0x00007f6ccbd1d000)
    libpython3.5m.so.1.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/python-3.5.1-5q5rsrtjld4u6jiicuvtnx52m7tfhegi/lib/libpython3.5m.so.1.0 (0x00007f6ccb7f7000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6ccb5db000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f6ccb219000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6ccc440000)
    libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f6ccb014000)
    libutil.so.1 => /usr/lib64/libutil.so.1 (0x00007f6ccae11000)
```

Same problem, it cannot load `libopenblas.so`.  So let's tell Spack to load it explicitly and try again:

```
[me@ankeli spack2]$ spack load openblas
[me@ankeli spack2]$ ldd `spack location -i py-numpy`/lib/python3.5/site-packages/numpy-1.11.0-py3.5-linux-x86_64.egg/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffc78d2c000)
    libopenblas.so.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/openblas-0.2.17-pw6rmlom7apfsnjtzfttyayzc7nx5e7y/lib/libopenblas.so.0 (0x00007fb61f48e000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fb61f175000)
    libpython3.5m.so.1.0 => /home/me/spack2/opt/spack/linux-x86_64/gcc-4.9.3/python-3.5.1-5q5rsrtjld4u6jiicuvtnx52m7tfhegi/lib/libpython3.5m.so.1.0 (0x00007fb61ec4f000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb61ea32000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fb61e671000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb620878000)
    libgfortran.so.3 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/libgfortran.so.3 (0x00007fb61e352000)
    libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fb61e14d000)
    libutil.so.1 => /usr/lib64/libutil.so.1 (0x00007fb61df4a000)
    libquadmath.so.0 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/../lib64/libquadmath.so.0 (0x00007fb61dd0c000)
    libgcc_s.so.1 => /home/me/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-4.9.3-layphctulnk3omsbjpzftqv6dlxpfe3d/lib64/../lib64/libgcc_s.so.1 (0x00007fb61daf6000)
```

OK, now it works with `ldd`.  And try with Python:

```
[me@ankeli spack2]$ python3 -c 'import numpy'
```

Remember... LD_LIBRARY_PATH is EVIL!!!



Discussion:
Another consequence of this phenomenon is that it is not possible to BUILD some Python packages without first LOADING their dependencies.  The following does NOT work:

```
spack build py-numpy
spack build py-netcdf     # setup.py cannot import numpy
```

At this point, I don't know how to get this working.

Using the develop branch I was able to build py-netcdf, using 'spack
install...', without any modules loaded or activated.

Glenn

On Fri, Apr 1, 2016 at 9:35 PM, Elizabeth Fischer notifications@github.com
wrote:

> Another consequence of this phenomenon is that it is not possible to BUILD
> some Python packages without first LOADING their dependencies. The
> following does NOT work:
> 
> spack build py-numpy
> spack build py-netcdf     # Build cannot import numpy
> 
> This modified version DOES work:
> 
> spack build py-numpy
> spack load py-numpy
> spack build py-netcdf     # Build cannot import numpy
> 
> or:
> 
> spack build py-numpy
> spack activate py-numpy
> spack build py-netcdf     # Build cannot import numpy
> 
> Should Spack load the modules of its dependencies when setting up a build
> environment?
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub
> https://github.com/LLNL/spack/issues/719#issuecomment-204633006

Thank you for the hint.  Looks like I've got my work cut out for me, digging through git diffs...
I will see if I can replicate your develop branch experience, building for Python3.

I tried the same thing from `develop` branch and it didn't work for me.

I think that SHA 220c72d6 (Merge branch 'features/blas-lapack-hardening' into develop) broke things here.  Before that commit, OpenBLAS ONLY built static libraries.  py-numpy would statically link OpenBLAS, and things would "just work."  Now, OpenBLAS builds shared by default.  py-numpy links to the shared OpenBLAS library (without RPATH) and things don't work.

An alternative workaround to the code posted in this PR would be to use openblas~shared.

PR #721 helps with this somewhat.  But it would be better if the new recursive module feature could list ONLY modules that are Python-related (i.e. that set PYTHONPATH in the module file).

I think I'm seeing the same problem as you. When I try to install py-scipy, I get:

```
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
```

This never happened to me before though. I wonder if something changed.

Load the scipy module
On Jul 6, 2016 12:17 PM, "Adam J. Stewart" notifications@github.com wrote:

> I think I'm seeing the same problem as you. When I try to install
> py-scipy, I get:
> 
> ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
> 
> This never happened to me before though. I wonder if something changed.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/LLNL/spack/issues/719#issuecomment-230823950, or mute
> the thread
> https://github.com/notifications/unsubscribe/AB1cdxNxneZODhuFaTG3AIGWm-LHf_dYks5qS9T-gaJpZM4H-DYi
> .

It's not a runtime problem, I literally can't even install it. I swear this worked a week or two ago. I wonder if some environment variables changed.

@adamjstewart you may want to check if [this](https://github.com/LLNL/spack/commit/31e9ded768b56d076eb452b56d1b5c94ac341761#diff-c8f14b8e4ac7661fb9ca8ecc00bc2ebc) or [this](https://github.com/LLNL/spack/commit/d687e332ad41cd7c84bf0007f8c331e56470da18#diff-c8f14b8e4ac7661fb9ca8ecc00bc2ebc) affected your build in any way

Just as a data point, I just successfully built py-scipy with openblas from the current spack devel branch.

@alalazo I commented out all of the changes from #1157 but it didn't help. I can't imagine that d687e33 is causing the problem since it only affects the PATH.

I also tried adding `depends_on('blas')` and `depends_on('lapack')` directly to `py-scipy`. It's almost like scipy is ignoring the `BLAS` and `LAPACK` environment variables we're setting.

@glennpj I just tried the latest develop but I'm still seeing the same problem. Here is the full spack-build.out:

```
==> '/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/python-2.7.11-voyxwv4ijmxs3qxyo5wjwxtps3uj2hhm/bin/python' 'setup.py' 'install' '--prefix=/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-scipy-0.17.0-wq7aypg6ggmfgiklptambydihsyxjqb7'
Traceback (most recent call last):
  File "setup.py", line 265, in <module>
    setup_package()
  File "setup.py", line 253, in setup_package
    from numpy.distutils.core import setup
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
```

Seems to be exactly what @citibeth was seeing. It's interesting that I'm only seeing it for the first time now though. @citibeth you mentioned that you had an idea of how to get RPATH working for Python modules?

@adamjstewart 
I have built packages with gcc-6.1 but I am not sure if I built the python stack with it. The successful build that I referenced was with the CentOS-7 system gcc (4.8.5). I will fire up a build with gcc-6.1 to see what happens.

@adamjstewart 

I can replicate what you are seeing with spack compiled gcc-6.1.0 and gcc-5.3.0. Building with the system gcc does work though.

Another consequence of this phenomenon is that it is not possible to BUILD some Python packages without first LOADING their dependencies.  The following does NOT work:

```
spack build py-numpy
spack build py-netcdf     # setup.py cannot import numpy
```

At this point, I don't know how to get this working.

Using the develop branch I was able to build py-netcdf, using 'spack
install...', without any modules loaded or activated.

Glenn

On Fri, Apr 1, 2016 at 9:35 PM, Elizabeth Fischer notifications@github.com
wrote:

> Another consequence of this phenomenon is that it is not possible to BUILD
> some Python packages without first LOADING their dependencies. The
> following does NOT work:
> 
> spack build py-numpy
> spack build py-netcdf     # Build cannot import numpy
> 
> This modified version DOES work:
> 
> spack build py-numpy
> spack load py-numpy
> spack build py-netcdf     # Build cannot import numpy
> 
> or:
> 
> spack build py-numpy
> spack activate py-numpy
> spack build py-netcdf     # Build cannot import numpy
> 
> Should Spack load the modules of its dependencies when setting up a build
> environment?
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub
> https://github.com/LLNL/spack/issues/719#issuecomment-204633006

Thank you for the hint.  Looks like I've got my work cut out for me, digging through git diffs...
I will see if I can replicate your develop branch experience, building for Python3.

I tried the same thing from `develop` branch and it didn't work for me.

I think that SHA 220c72d6 (Merge branch 'features/blas-lapack-hardening' into develop) broke things here.  Before that commit, OpenBLAS ONLY built static libraries.  py-numpy would statically link OpenBLAS, and things would "just work."  Now, OpenBLAS builds shared by default.  py-numpy links to the shared OpenBLAS library (without RPATH) and things don't work.

An alternative workaround to the code posted in this PR would be to use openblas~shared.

PR #721 helps with this somewhat.  But it would be better if the new recursive module feature could list ONLY modules that are Python-related (i.e. that set PYTHONPATH in the module file).

I think I'm seeing the same problem as you. When I try to install py-scipy, I get:

```
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
```

This never happened to me before though. I wonder if something changed.

Load the scipy module
On Jul 6, 2016 12:17 PM, "Adam J. Stewart" notifications@github.com wrote:

> I think I'm seeing the same problem as you. When I try to install
> py-scipy, I get:
> 
> ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
> 
> This never happened to me before though. I wonder if something changed.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/LLNL/spack/issues/719#issuecomment-230823950, or mute
> the thread
> https://github.com/notifications/unsubscribe/AB1cdxNxneZODhuFaTG3AIGWm-LHf_dYks5qS9T-gaJpZM4H-DYi
> .

It's not a runtime problem, I literally can't even install it. I swear this worked a week or two ago. I wonder if some environment variables changed.

@adamjstewart you may want to check if [this](https://github.com/LLNL/spack/commit/31e9ded768b56d076eb452b56d1b5c94ac341761#diff-c8f14b8e4ac7661fb9ca8ecc00bc2ebc) or [this](https://github.com/LLNL/spack/commit/d687e332ad41cd7c84bf0007f8c331e56470da18#diff-c8f14b8e4ac7661fb9ca8ecc00bc2ebc) affected your build in any way

Just as a data point, I just successfully built py-scipy with openblas from the current spack devel branch.

@alalazo I commented out all of the changes from #1157 but it didn't help. I can't imagine that d687e33 is causing the problem since it only affects the PATH.

I also tried adding `depends_on('blas')` and `depends_on('lapack')` directly to `py-scipy`. It's almost like scipy is ignoring the `BLAS` and `LAPACK` environment variables we're setting.

@glennpj I just tried the latest develop but I'm still seeing the same problem. Here is the full spack-build.out:

```
==> '/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/python-2.7.11-voyxwv4ijmxs3qxyo5wjwxtps3uj2hhm/bin/python' 'setup.py' 'install' '--prefix=/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-scipy-0.17.0-wq7aypg6ggmfgiklptambydihsyxjqb7'
Traceback (most recent call last):
  File "setup.py", line 265, in <module>
    setup_package()
  File "setup.py", line 253, in setup_package
    from numpy.distutils.core import setup
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/blues/gpfs/home/software/spack/opt/spack/linux-centos6-x86_64/gcc-6.1.0/py-numpy-1.11.0-j3gy5iw3op3zrpjnxzdu4c7vf6ocbojy/lib/python2.7/site-packages/numpy-1.11.0-py2.7-linux-x86_64.egg/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory
```

Seems to be exactly what @citibeth was seeing. It's interesting that I'm only seeing it for the first time now though. @citibeth you mentioned that you had an idea of how to get RPATH working for Python modules?

@adamjstewart 
I have built packages with gcc-6.1 but I am not sure if I built the python stack with it. The successful build that I referenced was with the CentOS-7 system gcc (4.8.5). I will fire up a build with gcc-6.1 to see what happens.

@adamjstewart 

I can replicate what you are seeing with spack compiled gcc-6.1.0 and gcc-5.3.0. Building with the system gcc does work though.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-numpy/package.py b/var/spack/repos/builtin/packages/py-numpy/package.py
--- a/var/spack/repos/builtin/packages/py-numpy/package.py
+++ b/var/spack/repos/builtin/packages/py-numpy/package.py
@@ -24,6 +24,7 @@
 ##############################################################################
 from spack import *
 
+
 class PyNumpy(Package):
     """NumPy is the fundamental package for scientific computing with Python.
     It contains among other things: a powerful N-dimensional array object,
@@ -38,7 +39,6 @@ class PyNumpy(Package):
     version('1.9.2',  'a1ed53432dbcd256398898d35bc8e645')
     version('1.9.1',  '78842b73560ec378142665e712ae4ad9')
 
-
     variant('blas',   default=True)
     variant('lapack', default=True)
 
@@ -63,6 +63,6 @@ def install(self, spec, prefix):
                 f.write('[DEFAULT]\n')
                 f.write('libraries=%s\n'    % ','.join(libraries))
                 f.write('library_dirs=%s\n' % ':'.join(library_dirs))
+                f.write('rpath=%s\n' % ':'.join(library_dirs))
 
         python('setup.py', 'install', '--prefix=%s' % prefix)
-


Test patch: