Repository: spack/spack

Issue:
Setting LD_LIBRARY_PATH to be "helpful" considered harmful
(with apologies to Djikstra and everyone else who's recycled that title meme)

**TL;DR**: the library that one of my spack binaries uses depends on what other spack packages I've `module load`-ed.  **YIKES**.  See also #3926.

I was trying to understand why @JusticeForMikeBrown was having trouble building bowtie2 (see #3950) when I've built it successfully with `gcc@4.x.y`.

His problem with `gcc@4.x.y` was zlib related; I checked the package and noticed that it doesn't have a dependency on zlib.  Perhaps it should, I thought.  Wonder what zlib my "production" copy was linked against?

```
$ ldd bowtie2-align-l  | grep libz
	libz.so.1 => /blah/spack/v0.0.8/opt/spack/linux-centos7-x86_64/gcc-5.4.0/zlib-1.2.11-ec535e2ikkpl7hd4y454t3yydjqorja6/lib/libz.so.1 (0x00002aaaaaf32000)
```

That surprised me, because there's no zlib dependency in the package.

Sure enough, it's because I have something else `module load`-ed that has the side effect of adding zlib's directory to `LD_LIBRARY_PATH`.

```
$ (unset LD_LIBRARY_PATH; ldd bowtie2-align-l) | grep libz
	libz.so.1 => /lib64/libz.so.1 (0x00002aaaaaf2f000)
```

My "newer" version of CentOS has a `/lib64/libz.so.1` that includes gzbuffer (`nm` didn't help, library's stripped...):

```
$ strings /lib64/libz.so.1 | grep buffer
gzbuffer
buffer error
```

so it (probably) works for me either way.

But imagine if there were two versions of a library (perhaps something mathematical) that give different results.  Now you have a program giving different results depending on what other Spack applications are also loaded.  

**THAT** would be fun to track down (assuming you even noticed...).

W.R.T. the main problem, bowtie2 should probably have a dependency on a new-ish version of zlib, but stuff like this is why LD_LIBRARY_PATH is a slippery tool to reach for.

I'll argue that this kind of unpredictability is a bigger negative than being helpful and always setting `LD_LIBRARY_PATH`.  This comment in the docs isn't actually correct:

> Spack avoids library misconfiguration by using RPATH to link dependencies. When a user links a library or runs a program, it is tied to the dependencies it was built with, so there is no need to manipulate LD_LIBRARY_PATH at runtime.
>
> [clipped from here](http://spack.readthedocs.io/en/latest/features.html?highlight=RPATH#packages-can-peacefully-coexist)

What would happen if `LD_LIBRARY_PATH` became *opt-in*, packages that need it specify it in their package definitions?

Looking at [the list of cases where RPATH support doesn't work](http://spack.readthedocs.io/en/latest/workflows.html?highlight=LD_LIBRARY_PATH#transitive-dependencies), it seems like 1) is not relevant (I think it's referring to `PERL5LIB`, etc...) and 3) are simply bugs.  That leaves 2), python extensions.  Is `RPATH` unworkable there or just not yet working?


Discussion:
I agree that the real problem here is with the missing `zlib` dependency for `bowtie2`. This is a great argument for why we need to add dependencies for everything we can, even if we expect it to be installed on every operating system ever.

> This comment in the docs isn't actually correct:
>
> > Spack avoids library misconfiguration by using RPATH to link dependencies. When a user links a library or runs a program, it is tied to the dependencies it was built with, so there is no need to manipulate LD_LIBRARY_PATH at runtime.

What's incorrect about that comment? As long as the dependency is stated, Spack will use RPATHs, making `LD_LIBRARY_PATH` unnecessary. We don't RPATH things in `/lib`, `/usr/lib64`, `/usr/local/lib`, etc. for the reasons stated in #3910, #2083, and the issues they fix. Basically, if `-Wl,-rpath,` is used with a system location, it can override Spack-built dependencies if it comes first. This is true for any directory that contains multiple installations, not just `/`, `/usr/`, and `/usr/local`.

> 1) is not relevant (I think it's referring to PERL5LIB, etc...)

Yes, it's referring to things like `PERL5LIB` and `PYTHONPATH`. Important, but not relevant to what you're worried about.

> That leaves 2), python extensions. Is RPATH unworkable there or just not yet working?

Unfortunately, Python doesn't really have any way of RPATHing dependencies. I don't know the full details, but @wscullin or @citibeth can explain more if you're interested.

> What would happen if `LD_LIBRARY_PATH` became _opt-in_, packages that need it specify it in their package definitions?

So there's two ways to think about this. You're thinking about it from the point of view of someone using the packages you installed. Yes, some packages don't work unless you have `LD_LIBRARY_PATH` set properly, but that's not the only reason our module files add it. Think about another scenario, in which you install some libraries like MPI and your users build _their own_ applications outside of Spack that link to your MPI libraries. If they don't know about `-Wl,-rpath`, their applications won't work unless `LD_LIBRARY_PATH` is set properly. Maybe we should add that use case to the documentation as well.

But, if you want it to be opt-in, you can do that. All you have to do is create a `modules.yaml` that looks like:
```yaml
modules:
  lmod:
    all:
      filter:
        environment_blacklist: 'LD_LIBRARY_PATH'
    bowtie2:
      environment:
        LD_LIBRARY_PATH: '$prefix/lib'
```
I haven't tested this, but it should work. See [Modules](http://spack.readthedocs.io/en/latest/module_file_support.html) and especially [Module Configuration Tutorial](http://spack.readthedocs.io/en/latest/tutorial_sc16_modules.html) for details.
[edited, jumping back to the top of my string of replies to @adamjstewart to make sure I'm not coming across a ranting maniac]

@adamjstewart made a bunch of points in the comment above and I've responded to them one by one.  I really don't see the value in this (even in automagically helping a set of programmers) and it does bother me.  I know it's not going to be the downfall of the known universe or Spack or ....

Hopefully the points (arguments?) that follow won't overshadow how valuable all of this is.

[end of edit]

---

> I agree that the real problem here is with the missing zlib dependency for bowtie2.

I think that will fix bowtie2's problem with the older gcc's.  I think that there is [perhaps also] a different issue with gcc@6.x.y.
> This is a great argument for why we need to add dependencies for everything we can, even if we expect it to be installed on every operating system ever.

So you're suggesting that every time someone adds a new library to spack that "someone" needs to audit every package in the tree that might use it and update them?

If it were *necessary* to set `LD_LIBRARY_PATH` then perhaps that would be reasonable, but since Spack did/does the [hard] work of using `RPATH` everywhere it can then the only justification is "in case someone wants to build against this library from outside of Spack.

An alternative would be to provide some documentation about using Spack libraries in non-Spack projects.
> What's incorrect about that comment?

Two things:

1. packages that use RPATH are only safe until someone adds a new library to Spack for them to trip over (e.g. bowtie2).  The fact that bowtie2 quietly depends on the system's zlib is only a bug (for those of us with modern-ish zlibs) because Spack sets `LD_LIBRARY_PATH` (though the quoted passage strongly suggests that it doesn't).  What happens when someone adds an updated copy of `libc` to the tree and everything in my world automagically picks it up.

     Spack includes OpenSSL and I recently skimmed past something where someone (@citibeth?) mentioned that Spack's best practice was for people to hardwire their system OpenSSL into their `packages.yaml` to avoid security surprises.  If I look at the package for the the software I want (perhaps something like `nginx` or the `shiny server` or ... where I need to start it as `root`) and it doesn't use OpenSSL, I might think that I was safe.

2.   This bit: "so there is no need to manipulate LD_LIBRARY_PATH at runtime".

     Because apparently there *is* a need.

3. (What are the two most common bugs in computer programs?  Off by one errors....) 

     That passage gave me false comfort.  I consider `LD_LIBRARY_PATH` slippery, dangerous, sloppy, Perl coders (and perhaps others) refer to this sort of thing as a "code smell".  I know that software's full of compromises and you all have helped me get things into the tree that are less than perfect (as always, **thank you**!).  If a package won't work without it and the package is important then so be it.  

      But one of the things that drew me to Spack was it being tight and clean, being nicely self contained/immune-to-its-neighbors *and* being able to put together a set of tools that aren't a hazard to the rest of the community.  The thing that I've replaced was an unprincipled mess of spaghetti and I was proud (by association) that I had found a better solution.

     I think that's why all of this is so surprising to me.


> Python

Spack breaks system utilities.

At first I thought that this was a `PYTHONPATH` thing, but it's `LD_LIBRARY_PATH`.

I understand that Python apps are a separate case (I wonder if the other dynamic languages are too, we just haven't noticed yet?), but....

```
$ yum --version
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 15 2017, 18:37:03)
[GCC 5.4.0]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq


$ unset LD_LIBRARY_PATH
$ yum --version
3.4.3
  Installed: rpm-4.11.3-17.el7.x86_64 at 2017-02-21 22:42
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2015-11-20 11:15
  Committed: Florian Festi <ffesti@redhat.com> at 2015-09-11

  Installed: subscription-manager-1.15.9-15.el7.centos.0.1.x86_64 at 2017-02-21 22:42
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2015-12-08 18:57
  Committed: Johnny Hughes <johnny@centos.org> at 2015-12-08

  Installed: yum-3.4.3-132.el7.centos.0.1.noarch at 2017-02-21 22:42
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2015-12-03 15:33
  Committed: Johnny Hughes <johnny@centos.org> at 2015-12-03

  Installed: yum-plugin-fastestmirror-1.1.31-34.el7.noarch at 2017-02-21 22:42
  Built    : CentOS BuildSystem <http://bugs.centos.org> at 2015-11-20 15:34
  Committed: Valentina Mukhamedzhanova <vmukhame@redhat.com> at 2015-10-12
```

Why subject a community to this unless we have to?
> Think about another scenario, in which you install some libraries like MPI and your users build their own applications outside of Spack that link to your MPI libraries. If they don't know about -Wl,-rpath, their applications won't work unless LD_LIBRARY_PATH is set properly.

Two things (avoiding the joke above):

1.  What is the ratio of users of Spack based application trees to people who don't know about `-Wl,rpath` who are compiling applications that depend on libraries in a Spack deployment?

     The first group is potentially a victim of spooky-action-at-a-distance, the second group is relying on it.

2. What happens to these applications that were built by someone who doesn't understand/control these dependencies when someone changes the set of modules that are being loaded?  *kerblooie*.

     I'd argue that making things automagically work isn't doing them any favors
> All you have to do is create a modules.yaml that looks like:

Yes, I can do that (mentioned playing with it in #3926).

The problem is that it puts the burden of figuring out which modules really need `LD_LIBRARY_PATH` and which don't on me.  I'm poorly suited for that.  The module authors (including me, at times) are in the best position to know what their packages *need* and to state it clearly in the package definition for all to see.  If they actually *need* `LD_LIBRARY_PATH`, that need should (sez me...) go in `setup_environment` or `setup_dependent_environment` and be vetted by the reviewers.

If I blacklist `LD_LIBRARY_PATH` everywhere, then I'm constantly going to be reacting to problems (probably reported to me by my users).

Package authors/updaters, on the other hand, know ahead of time.


> If it were necessary to set LD_LIBRARY_PATH then perhaps that would be reasonable, but since Spack did/does the [hard] work of using RPATH everywhere it can then the only justification is "in case someone wants to build against this library from outside of Spack.

Let me add a few words to your description: `LD_LIBRARY_PATH` is set in module files "in case someone wants to build against this library from outside of Spack, and doesn't necessarily know that Spack exists". 

Users on a cluster may be unaware of how you built the software that is available through module files, and I don't think telling them to add several lines of directories with strange hashes to manually set the RPATH will help.

> An alternative would be to provide some documentation about using Spack libraries in non-Spack projects.

I think this will be what the [environment project](https://github.com/LLNL/spack/wiki/Spack-Environments-WG) will try to achieve: set a development environment managed by Spack. And that means also that Spack will take care of setting RPATHs for you.
> What happens when someone adds an updated copy of libc to the tree and everything in my world automagically picks it up.

I see the point here. If I remember well there was a similar discussion sometimes ago (in a PR that was meant to introduce `libc`, but was never finished). 

My personal point of view is that the current state in Spack is a trade off between usability and theoretical correctness, and we should rely on whatever `libc` (or other similar kind of low level libraries) we have in our OS. We must stop at some point in the software stack, otherwise to be completely isolated from the environment where we install software we'll  turn the Spack PM project into the Spack OS project. 

> This bit: "so there is no need to manipulate LD_LIBRARY_PATH at runtime".

There should be no need if you use a Spack built application. If you use a Spack built library things must be put in perspective: you don't need to set search paths for transitive dependencies (they are RPATHed). So maybe "so there is no need to manipulate LD_LIBRARY_PATH at runtime for the software that was built by Spack" sounds better?

> > 1) is not relevant (I think it's referring to PERL5LIB, etc...)
>
> Yes, it's referring to things like `PERL5LIB` and `PYTHONPATH`. Important, but not relevant to what you're worried about.

PYTHONPATH is evil for the same reason LD_LIBRARY_PATH is evil.  I did think up a proposal on how to get RPATH-like functionality in Python --- basically edit every .py file when it's being installed, to set sys.path correctly for imports.  Or someting like that.  It should work in theory, but someone would have to do it and find all the corner cases.

In the meantime, PYTHONPATH isn't quite as evil as LD_LIBRARY_PATH because Python linking is less evil.  If you run with py-numpy@1.1.2 instead of py-numpy@1.1.1, probably nothing bad will happen.  But if you dynamically link to the wrong version of a shared lib, the program will probably crash.

> > That leaves 2), python extensions. Is RPATH unworkable there or just not yet working?
> 
> Unfortunately, Python doesn't really have any way of RPATHing dependencies. I don't know the full details, but @wscullin or @citibeth can explain more if you're interested.

Yes that's the end result, see #719 for background.  This is probably fixable, but someone would have to look more closely at Python's setuptools and distutils, and figure out how to do it right.  In the meantime, Python extensions need LD_LIBRARY_PATH to run properly.

> your users build _their own_ applications outside of Spack that link to your MPI libraries. If they don't know about `-Wl,-rpath`, their applications won't work unless `LD_LIBRARY_PATH` is set properly. Maybe we should add that use case to the documentation as well.

Everyone should build their applications in Spack.  Yes, I'm mostly serious.  Some more work on Spack Environments, and we might be closer to that Nirvana.

FWIW, I actually relate to what @hartzell's referring to here, and I wouldn't be completely opposed to removing `LD_LIBRARY_PATH` from modules by default, if we documented it sufficiently.  I *think* most module systems rely on it, though, and I think most HPC users will expect it in their modules.  So maybe we should call more attention to how to *unset* this.

I think the answer here is to put in more checks to ensure that we're actually RPATH-ing all the dependencies in Spack builds, as that will at least ensure that Spack-built binaries work.
@tgamblin --

> I think most module systems rely on it, though

When you say "module systems", do you mean creatures like Lmod?

I can see where the environments which e.g. Lmod or Tcl-modules are trying to set up aren't built with RPATH-ing and depend on `LD_LIBRARY_PATH` but I can't see how Lmod or Tcl-modules inherently need it.

Would a more nit-picky wording of your statement be: "most HPC computing enviroments rely on it" or some such (putting the blame on how the apps were built, not on the tool being used to provision the user's environment for them)?

> I think most HPC users will expect it in their modules.

If a) always setting `LD_LIBRARY_PATH` is useful because it helps people build software and b) "most HPC users will expect it" does that mean that most HPC users are building software?

Most of my users are not building their own software (be careful what you wish for, I've been told, you might get it...), which might be why I seem to be landing in the minority.
@hartzell 

I cannot replicate your yum issue:

```
[root@node103 spack]# spack load python@3.5.2
[root@node103 spack]# spack load bowtie2
[root@node103 spack]# spack load gcc@6.3.0
[root@node103 spack]# yum --version
3.2.29

Unable to read consumer identity
  Installed: rpm-4.8.0-27.el6.x86_64 at 2014-09-22 01:19
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at 2012-04-04 08:43
  Committed: Panu Matilainen <pmatilai@redhat.com> at 2012-04-04

  Installed: subscription-manager-0.99.19-1.el6.x86_64 at 2014-09-22 01:27
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at 2012-05-22 19:19
  Committed: Devan Goodwin <dgoodwin@rm-rf.ca> at 2012-05-22

  Installed: yum-3.2.29-30.el6.noarch at 2014-09-22 01:20
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at 2012-05-09 11:42
  Committed: Zdenek Pavlas <zpavlas@redhat.com> at 2012-05-09

  Installed: yum-plugin-aliases-1.1.30-14.el6.noarch at 2014-09-22 01:32
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at 2012-04-26 11:35
  Committed: Zdenek Pavlas <zpavlas@redhat.com> at 2012-04-26

[root@node103 spack]# echo $LD_LIBRARY_PATH 
/pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/gcc-6.3.0-han2ovjpjdnuwkhtfhugdq66cjkte7q3/lib64:/pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/gcc-6.3.0-xnoykzftyh27xs6vqkwvtft2cce5yd4k/lib64

```

I do not think glibc should be added to spack unless there's extensive testing and it's proven to not break things.  If users need a newer version of glibc then I think the better solution would be to build a separate spack environment against the newer operating system which has that newer version of glibc.  For example I have separate spack environments for centos 6 and centos 7 servers.  

To allow users to switch between the separate operating system environments I tell them to put this in their bashrc:


```
# detect kernel then source spack file if it exists
KERNEL=$(uname -r | cut -d '.' -f1,2)

if [ "${KERNEL}" = "2.6" ] ; then
        if [ -f /pbtech_mounts/softlib001/apps/EL6/spack/share/spack/setup-env.sh ] ; then
                . /pbtech_mounts/softlib001/apps/EL6/spack/share/spack/setup-env.sh
        fi
fi

if [ "${KERNEL}" = "3.10" ] ; then
        if [ -f /pbtech_mounts/softlib001/apps/EL7/spack/share/spack/setup-env.sh ] ; then
                . /pbtech_mounts/softlib001/apps/EL7/spack/share/spack/setup-env.sh
        fi
fi

```


Moreover, if you need a very new version of glibc, one could try out something like containers as we've been using Singularity http://singularity.lbl.gov then use spack within that.  We haven't tested yet but I think it would work.




> I cannot replicate your yum issue:
> 
> [root@node103 spack]# spack load python@3.5.2
> [...]

I suspect that's because yum uses the system Python, which is Python2.

```
hartzelg@lb097hmdev:~$ head /usr/bin/yum
#!/usr/bin/python
import sys
try:
```

The Python you tested with is @3.5.2, and presumably whatever thing that's tripping yum up is sufficiently different.

I can replicate it fairly easily:

```
hartzelg@lb097hmdev:~$ module purge
hartzelg@lb097hmdev:~$ yum |& head
Loaded plugins: fastestmirror, package_upload, priorities, product-id, search-
              : disabled-repos, subscription-manager
You need to give some command
Usage: yum [options] COMMAND

List of Commands:

check          Check for problems in the rpmdb
check-update   Check for available package updates
clean          Remove cached data
hartzelg@lb097hmdev:~$ module load gcc
hartzelg@lb097hmdev:~$ module load python/2.7.13
hartzelg@lb097hmdev:~$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 23 2017, 09:47:24)
[GCC 5.4.0]

If you cannot solve this problem yourself, please go to
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

```

Can you try it with a `python@2`?

g.

> I do not think glibc should be added to spack

I'm starting to regret using the example of `glibc`.  It's a worst-case scenario and I spent a lot of time chasing problems in linuxbrew related to building it but it's not categorically different than other packages that Spack installs.

The issue of being surprised by dynamically linking things that you didn't expect is real in other cases, if less catastrophic.
Thanks for the discussion!  I think that I understand the pros and cons.
@hartzell 

Discussion is fun!

Anyway, you closed but still cannot replicate on RHEL 6.3 node.

```
user@node103[~/repos/spack]$ which python
/usr/bin/python

user@node103[~/repos/spack]$ python -V
Python 2.6.6

user@node103[~/repos/spack]$ spack load python@2.7.13
                                                                                                                                                                                      
user@node103[~/repos/spack]$ python -V                                                                                                                                                                                                     
Python 2.7.13

user@node103[~/repos/spack]$ yum
Loaded plugins: aliases, changelog, downloadonly, kabi, presto, product-id, refresh-packagekit, security, subscription-manager, tmprepo, verify, versionlock
Not root, certificate-based repositories not updated
Loading support for Red Hat kernel ABI
You need to give some command
Usage: yum [options] COMMAND

user@node103[~/repos/spack]$ echo $LD_LIBRARY_PATH 
/pbtech_mounts/homes027/user/repos/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/python-2.7.13-wyawocw4q6fthbhfbmoe6uecx4njg76e/lib
```


Hmmm.  I wonder if the difference is my `module load` (via Lmod) vs. your `spack load`?

Do you end up with `/.../opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib` on your `LD_LIBRARY_PATH`?  (hash might differ)?

I do, and if I surgically delete just that directory from the `LD_LIBRARY_PATH` then `yum` works again.
@hartzell 

I do 

Old Centos 6.3 node:

```
user@node103[~/repos/spack]$ echo $LD_LIBRARY_PATH 
/pbtech_mounts/homes027/dod2014/repos/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/python-2.7.13-wyawocw4q6fthbhfbmoe6uecx4njg76e/lib
```

```
Python 2.7.13 (default, May 11 2017, 15:07:28) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
>>> import sys
>>> import yum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named yum
```

So it seems that the Spack-provided Python indeed will break yum.  I don't understand though why mine's while your's isn't.  

Though I do now have the same problem on Centos 7.3 box.

```
user@france[~]$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 25 2017, 10:47:26) 
[GCC 6.3.0]

If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq
  


echo $LD_LIBRARY_PATH 
/pbtech_mounts/softlib001/apps/EL7/spack/opt/spack/linux-centos7-x86_64/gcc-6.3.0/python-2.7.13-j34ptvqierishxtfdndhf4n2ia6vk55v/lib:/pbtech_mounts/softlib001/apps/EL7/spack/opt/spack/linux-centos7-x86_64/gcc-6.3.0/python-2.7.13-vb7ici6xj2dni5voyan5kotc7yilgi2y/lib:/pbtech_mounts/softlib001/apps/EL7/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/python-2.7.13-hrkwpc3zx6736aapgstxuatwdm5vhszs/l

user@france[~]$ spack load python@2.7 %gcc@4.8.5

user@france[~]$ python -V
Python 2.7.13

user@france[~]$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, May 16 2017, 11:15:46) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq


```

It could be that one was RPATHed and one wasn't. But I'm just speculating.
If I run `strace yum` after loading python@2.7.13, it gives up after trying to load `yum.{so,py,pyc}|yummodule.so`.

```
stat("/usr/bin", {st_mode=S_IFDIR|0555, st_size=36864, ...}) = 0
stat("/usr/bin", {st_mode=S_IFDIR|0555, st_size=36864, ...}) = 0
stat("/usr/bin/yum", {st_mode=S_IFREG|0755, st_size=801, ...}) = 0
open("/usr/bin/yum.so", O_RDONLY)       = -1 ENOENT (No such file or directory)
open("/usr/bin/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/bin/yum.py", O_RDONLY)       = -1 ENOENT (No such file or directory)
open("/usr/bin/yum.pyc", O_RDONLY)      = -1 ENOENT (No such file or directory)
stat("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yum", 0x7ffffffeaed0) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/httpie-0.9.8-k27pwfdxo3wumv2nbtmp4n7sjhl2qy7s/lib/python2.7/site-packages/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-34.4.1-wuvgblqg2ttm7qar7x43qun5cyi356i5/lib/python2.7/site-packages/yum", 0x7ffffffeaed0) = -1 ENOENT (No such file or directory)
[...]
stat("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yum", 0x7ffffffeaed0) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-e4mcx4jdyucgc2fiqz35dpezmxfp6o2f/lib/python2.7/site-packages/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "There was a problem importing on"..., 500There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named yum

Please install a package which provides this module, or
```

I've trimmed the above, but trust me that it never looks for it in any of the system directories.

If I `unset LD_LIBRARY_PATH` then `strace yum` then it runs successfully (here's the analogous bit to the above.  Note that it finds what it's looking for in `/usr/lib/python2.7/site-packages/yum`.

```
stat("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sc1/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-pysocks-1.6.6-cmakuuvdpuktjn2m4rdfr4kh2p32nxlf/lib/python2.7/site-packages/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/plat-linux2/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/plat-linux2/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/lib-dynload/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/lib-dynload/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/site-packages/yum", 0x7ffffffec760) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/site-packages/yum.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/site-packages/yummodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/site-packages/yum.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/python2.7/site-packages/yum.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/python2.7/site-packages/yum", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/lib/python2.7/site-packages/yum/__init__.py", {st_mode=S_IFREG|0755, st_size=305199, ...}) = 0
stat("/usr/lib/python2.7/site-packages/yum/__init__", 0x7ffffffec710) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/site-packages/yum/__init__.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/site-packages/yum/__init__module.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/site-packages/yum/__init__.py", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=305199, ...}) = 0
open("/usr/lib/python2.7/site-packages/yum/__init__.pyc", O_RDONLY) = 4
```

Without LD_LIBRARY_PATH to distract it, it looks in the system dirs and finds what it needs.

Why would setting LD_LIBRARY_PATH (used to search for shared libraries) influence Python's search for a python package?
@hartzell @adamjstewart

On Centos 7.3 the rpm yum-3.4.3-150.el7.centos.noarch contains files within:

```
user@france[~]$ rpm -ql yum-3.4.3-150.el7.centos.noarch | grep py | head
/usr/lib/python2.7/site-packages/rpmUtils
/usr/lib/python2.7/site-packages/rpmUtils/__init__.py
/usr/lib/python2.7/site-packages/rpmUtils/__init__.pyc
/usr/lib/python2.7/site-packages/rpmUtils/arch.py
/usr/lib/python2.7/site-packages/rpmUtils/arch.pyc
/usr/lib/python2.7/site-packages/rpmUtils/miscutils.py
/usr/lib/python2.7/site-packages/rpmUtils/miscutils.pyc
/usr/lib/python2.7/site-packages/rpmUtils/oldUtils.py
/usr/lib/python2.7/site-packages/rpmUtils/oldUtils.pyc
/usr/lib/python2.7/site-packages/rpmUtils/transaction.py

user@france[~]$ export PYTHONPATH=/usr/lib/python2.7/site-packages

user@france[~]$ echo $PYTHONPATH 
/usr/lib/python2.7/site-packages

user@france[~]$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   No module named rpm

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 25 2017, 10:47:26) 
[GCC 6.3.0]

If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

```

```
user@france[~]$ rpm -ql rpm-python-4.11.3-21.el7.x86_64 | grep -e rpm | head                                                                                                                                                               
/usr/lib64/python2.7/site-packages/rpm
/usr/lib64/python2.7/site-packages/rpm/__init__.py
/usr/lib64/python2.7/site-packages/rpm/__init__.pyc
/usr/lib64/python2.7/site-packages/rpm/__init__.pyo
/usr/lib64/python2.7/site-packages/rpm/_rpm.so
/usr/lib64/python2.7/site-packages/rpm/_rpmb.so
/usr/lib64/python2.7/site-packages/rpm/_rpms.so
/usr/lib64/python2.7/site-packages/rpm/transaction.py
/usr/lib64/python2.7/site-packages/rpm/transaction.pyc
/usr/lib64/python2.7/site-packages/rpm/transaction.py

```

```
user@france[~]$ export PYTHONPATH=/usr/lib/python2.7/site-packages:/usr/lib64/python2.7/site-packages

user@france[~]$ echo $PYTHONPATH 
/usr/lib/python2.7/site-packages:/usr/lib64/python2.7/site-packages

user@france[~]$ yum
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:

   /usr/lib64/python2.7/site-packages/rpm/_rpm.so: undefined symbol: PyUnicodeUCS4_AsUTF8String

Please install a package which provides this module, or
verify that the module is installed correctly.

It's possible that the above module doesn't match the
current version of Python, which is:
2.7.13 (default, Apr 25 2017, 10:47:26) 
[GCC 6.3.0]

If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

```


So it seems to be an issue with the fact that yum's python things are not installed within Spack's Python environment.

I don't consider this to be a problem as our users are not using yum though it would be nice if we could fix somehow since I source Spack via ~/.bashrc.  Though I always do "sudo yum" so not, actually...
Nice digging.

I'm content to close this (again).  There's enough documentation here if any ever trips over it and it's not a day-to-day problem (yet).


Here's another cute one (CentOS 7), where Spack's *libtiff* is screwing up the system's `ps2pdf` application:

```
hartzelg@blah:spack-graphviz (bug/graphviz-and-swig)$ ps2pdf foo.gv.ps2 foo.pdf
/usr/bin/gs: /doh/apps/spack/v0.0.10/opt/spack/linux-centos7-x86_64/gcc-5.4.0/libtiff-4.0.6-g7gpsqww4hfkg5tg4nr5izmc2sj76c2z/lib/libtiff.so.5: no version information available (required by /lib64/libgs.so.9)
hartzelg@blah:spack-graphviz (bug/graphviz-and-swig)$ unset LD_LIBRARY_PATH; ps2pdf foo.gv.ps2 foo.pdf
hartzelg@blah:spack-graphviz (bug/graphviz-and-swig)$ file foo.pdf
foo.pdf: PDF document, version 1.4
```

I think that I'll take a test drive of @adamjstewart's blacklisting suggestion and see what breaks.



> I think that I'll take a test drive of @adamjstewart's blacklisting suggestion and see what breaks.

Nothing will break, but your modules might not be as easy to use. If users builds their software outside of Spack and link to a library you installed, their applications will not work unless:

1. They know about and use RPATHs, or
2. They set `LD_LIBRARY_PATH` themselves, requiring them to know where Spack installed it

This is the real reason that we set `LD_LIBRARY_PATH` in the module files.
> Nothing will break

Wow, an optimist!  :smile_cat:

I think that we established earlier in the thread that we support a variety of user communities with different needs/priorities.

I have many inexperienced users trying to use the systems I assemble.  Having `/usr/bin/ps2pdf` fail for something they've never heard of is very hard to support.

I have a few users building software against the systems I assemble.  I can fairly effectively argue that variants of the words "software developer" in their job titles mean that I can expect them to step up and learn how things work.  I'm uncomfortable with locally built software magically (transparently) depending on my trees for the same reason I'm uncomfortable with Spack packages magically leaning on system libraries; I like dependencies to be explicit.  **YMMV**...


You're lucky if you get to work with "software developers". A surprising amount of my users have never even used Linux before. Not sure who gave them permission to use a super computer...
I stumbled over this, because LD_LIBRARY_PATH breaks nearly anything using ncurses on my Debian-10 machine:

```console
$ /usr/bin/clear_console
/usr/bin/clear_console: /opt/spack/install-tree/debian10/gcc-9.3.0/ncurses-6.2-zqozh/lib/libtinfo.so.6: no version information available (required by /usr/bin/clear_console)
```

So I would like to blacklist LD_LIBRARY_PATH for `spack load`. I just haven't found an easy way to do that?

And on the other side, modules.yaml / prefix_inspections doesn't seem to be honored.
I'm seeing exactly the same issue, on Debian stable (10) and testing/bullseye (11), I'm getting:
```
bash: /home/gitlab-runner/builds/c-5dWVoW/0/cw-group/cw-software/.spack/opt/spack/linux-debiantesting-x86_64/gcc-10.1.0/ncurses-6.2-g663epv4unoj2sohfwgpumxnyvvnverl/lib/libtinfo.so.6: no version information available (required by bash)
```

which seems to come from `spack load ncurses`, as
```
$ echo $LD_LIBRARY_PATH

$ ldd /bin/bash
	linux-vdso.so.1 (0x00007ffc33317000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f156e59a000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f156e595000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f156e3d4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f156e734000)
$ spack load ncurses
$ ldd /bin/bash
/bin/bash: /home/repr/cw-software-0.15/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/ncurses-6.2-mkf5o7z3jz7zm4vyylvlawgdbvry5zzj/lib/libtinfo.so.6: no version information available (required by /bin/bash)
	linux-vdso.so.1 (0x00007ffcf359d000)
	libtinfo.so.6 => /home/repr/cw-software-0.15/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/ncurses-6.2-mkf5o7z3jz7zm4vyylvlawgdbvry5zzj/lib/libtinfo.so.6 (0x00007f06faecd000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f06fae86000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f06facc5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f06fb036000)
$ echo $LD_LIBRARY_PATH
/home/repr/cw-software-0.15/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/ncurses-6.2-mkf5o7z3jz7zm4vyylvlawgdbvry5zzj/lib:/home/repr/cw-software-0.15/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/ncurses-6.2-mkf5o7z3jz7zm4vyylvlawgdbvry5zzj/lib
```

So `spack load` sets `LD_LIBRARY_PATH`, which then 'breaks' system tools. Is there any easy way to turn off LD_LIBRARY_PATH setting in `spack load` ?



If https://github.com/spack/spack/issues/17457#issuecomment-657473823 would get implemented, you could override the `prefix_inspections` in modules.yaml.

Until then I see two workarounds:
* Don't use `spack load` and use normale environment modules / lmod: Let spack generate them for you, then use `module load foo`.
* Patch `lib/spack/spack/user_environment.py:prefix_inspections()` in your local spack.

I might look into implementing https://github.com/spack/spack/issues/17457#issuecomment-657473823 sometime.
Another question that lately came up concerning this topic:

Let's assume one disables `LD_LIBRARY_PATH` globally (as described above in various ways).
Let's further assume doing so is officially supported.

How does the `package.py` for a specific package that *really* needs `LD_LIBRARY_PATH` to be set (for whatever reason), and knows, that it will likely not break too much, go at setting it?
* Would it simply do that in `setup_*_environment`?
* Do we have another method for this?
Since https://github.com/spack/spack/issues/18689 was closed as a duplicate, let me add the tl;dr here:

In the last two weeks two people reported issues on Slack where `spack load` broke system executables as a result of spack paths in LD_LIBRARY_PATH.

In particular: `spack load cmake` causes system `git` to fail on CentOS 8, because openssl gets pirated.

Another instance is where my terminal/shell became completely unusable after `spack load [dependent of ncurses]` as system lua started using an in compatible ncurses provided by spack, resulting in a warning message at the start of every line, as I'm using [z.lua](https://github.com/skywind3000/z.lua) in my shell. So no spack load for me :(

---

I would suggest to not modify `LD_LIBRARY_PATH` upon `spack load [...]`, except when the user explicitly asks for it: `spack load --libraries [...]` or so.
Just to add another point to this, many Python packages don't support RPATH, and won't work unless `(DY)?LD_LIBRARY_PATH` is set.
> Python packages don't support RPATH

Does Python maybe have a dedicated env variable for search paths it dlopens?

If not, how about adding a prop `export_search_paths = false` to PackageBase, and set it to `true` for `PythonPackage`? Then `spack load` would only have to add paths to `(DY)LD_LIBRARY_PATH` whenever `export_search_paths` is true.
> Does Python maybe have a dedicated env variable for search paths it dlopens?

Not that I know of, just `(DY)?LD_LIBRARY_PATH`.
Personally I would prefer that LD_LIBRARY_PATH always be prepended with useful shared libs, when available.  Yes, it can cause trouble if you're trying to mix spack and non-spack packages.  But, if you do that, you're already living in sin--don't do it.

A middle ground would be to add a flag to 'spack load' that would optionally also export libs this way.  So, they could be exported when needed, but not by default.

If nothing else, it would be handy to (optionally?) set a package-specific env var so that the lib location in question could be easily derived, rather than having to determine it by magical means.
> Since #18689 was closed as a duplicate, let me add the tl;dr here:
> 
> In the last two weeks two people reported issues on Slack where `spack load` broke system executables as a result of spack paths in LD_LIBRARY_PATH.
> 
> In particular: `spack load cmake` causes system `git` to fail on CentOS 8, because openssl gets pirated.
> 
> Another instance is where my terminal/shell became completely unusable after `spack load [dependent of ncurses]` as system lua started using an in compatible ncurses provided by spack, resulting in a warning message at the start of every line, as I'm using [z.lua](https://github.com/skywind3000/z.lua) in my shell. So no spack load for me :(
> 
> I would suggest to not modify `LD_LIBRARY_PATH` upon `spack load [...]`, except when the user explicitly asks for it: `spack load --libraries [...]` or so.

`git` also does not work after `spack load openmpi` on CentOS 8 :(

```
bwibking@avatargpue:~/quokka> ldd /usr/libexec/git-core/git-remote-https
	linux-vdso.so.1 (0x00007ffcb07d3000)
	libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f6fefcfa000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f6fefabf000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f6fef83b000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f6fef624000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6fef404000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f6fef1fc000)
	libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f6feed16000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f6fee951000)
	libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f6fee72a000)
	libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f6fee50c000)
	libssh.so.4 => /lib64/libssh.so.4 (0x00007f6fee29d000)
	libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f6fee08c000)
	libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f6feddf8000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f6fedba3000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f6fed8ba000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f6fed6a3000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f6fed49f000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f6fed251000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f6fed041000)
	libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f6fece34000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6ff0385000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f6fecc30000)
	libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f6fec8af000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f6fec69e000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6fec49a000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6fec283000)
	libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f6fec065000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f6febce3000)
	libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f6febac2000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f6feb898000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f6feb66f000)
bwibking@avatargpue:~/quokka> spack load openmpi
bwibking@avatargpue:~/quokka> ldd /usr/libexec/git-core/git-remote-https
	linux-vdso.so.1 (0x00007ffea5ae9000)
	libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f4b7b7d2000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f4b7b597000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f4b7b313000)
	libz.so.1 => /avatar/bwibking/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-8.4.1/zlib-1.2.11-hlaky7kmtfaieud3deytwmi73gkicoby/lib/libz.so.1 (0x00007f4b7b0fc000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4b7aedc000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f4b7acd4000)
	libcrypto.so.1.1 => /avatar/bwibking/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-8.4.1/openssl-1.1.1l-2k7gnwjtptaejuxphbnvwn7s6lm3szp6/lib/libcrypto.so.1.1 (0x00007f4b7a7ee000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f4b7a429000)
	libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f4b7a202000)
	libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f4b79fe4000)
	libssh.so.4 => /lib64/libssh.so.4 (0x00007f4b79d75000)
	libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f4b79b64000)
	libssl.so.1.1 => /avatar/bwibking/spack/opt/spack/linux-rocky8-skylake_avx512/gcc-8.4.1/openssl-1.1.1l-2k7gnwjtptaejuxphbnvwn7s6lm3szp6/lib/libssl.so.1.1 (0x00007f4b798d2000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f4b7967d000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f4b79394000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f4b7917d000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f4b78f79000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f4b78d2b000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f4b78b1b000)
	libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f4b7890e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4b7be5d000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f4b7870a000)
	libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f4b78389000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f4b78178000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f4b77f74000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f4b77d5d000)
	libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f4b77b3f000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f4b777bd000)
	libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f4b7759c000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f4b77372000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f4b77149000)
```
Unfortunately, `spack load cmake`, `spack load openmpi` and `spack load rsync` all break git on CentOS / Rocky 8 :(

Any workarounds would be appreciated!
1. I generally use spack environments not spack load.  Either with views or
modules.

2. In assembling the environment I generally address issues like this as
the come up.  In this case I would save LD-Library-Path then source all the
module loads, then restore ld library path.  That would all happen in the
final environment loading script I produce.

3. In general I would use the spack command to build and assemble
environments but not to load them.

On Thu, Dec 23, 2021 at 23:56 Ben Wibking ***@***.***> wrote:

> Unfortunatley, spack load cmake, spack load openmpi and spack load rsync
> all break git on CentOS / Rocky 8 :(
>
> Any workarounds would be appreciated!
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/spack/spack/issues/3955#issuecomment-1000654269>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAOVY52NKYOZO4GUGQXBV6LUSP4QZANCNFSM4DIUO3HQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
-- 
Elizabeth Ashley Fischer, she/her
University of Alaska, Geophysical Institute
cell: 617-308-0436

UA is an AA/EO employer and educational institution and prohibits illegal
discrimination against any individual: www.alaska.edu/nondiscrimination.  UAF
is located at Troth Yeddha', on the traditional homelands of the Lower
Tanana Dené People and UAF therefore benefits from the long stewardship of
these lands.

@BenWibking Can you try using the following `modules.yaml`?
```yaml
modules:
  prefix_inspections::  # notice the double colon
    bin:
      - PATH
    man:
      - MANPATH
    share/man:
      - MANPATH
    share/aclocal:
      - ACLOCAL_PATH
    lib/pkgconfig:
      - PKG_CONFIG_PATH
    lib64/pkgconfig:
      - PKG_CONFIG_PATH
    share/pkgconfig:
      - PKG_CONFIG_PATH
    '':
      - CMAKE_PREFIX_PATH
```
This will override the default settings and prevent Spack from setting `LD_LIBRARY_PATH`. I know this works for module file creation. If it doesn't work for `spack load`, let me know.
It works, thanks!

Resolving patch:
diff --git a/lib/spack/spack/user_environment.py b/lib/spack/spack/user_environment.py
--- a/lib/spack/spack/user_environment.py
+++ b/lib/spack/spack/user_environment.py
@@ -32,12 +32,9 @@ def prefix_inspections(platform):
 
     inspections = {
         "bin": ["PATH"],
-        "lib": ["LD_LIBRARY_PATH", "LIBRARY_PATH"],
-        "lib64": ["LD_LIBRARY_PATH", "LIBRARY_PATH"],
         "man": ["MANPATH"],
         "share/man": ["MANPATH"],
         "share/aclocal": ["ACLOCAL_PATH"],
-        "include": ["CPATH"],
         "lib/pkgconfig": ["PKG_CONFIG_PATH"],
         "lib64/pkgconfig": ["PKG_CONFIG_PATH"],
         "share/pkgconfig": ["PKG_CONFIG_PATH"],
@@ -45,8 +42,8 @@ def prefix_inspections(platform):
     }
 
     if platform == "darwin":
-        for subdir in ("lib", "lib64"):
-            inspections[subdir].append("DYLD_FALLBACK_LIBRARY_PATH")
+        inspections["lib"] = ["DYLD_FALLBACK_LIBRARY_PATH"]
+        inspections["lib64"] = ["DYLD_FALLBACK_LIBRARY_PATH"]
 
     return inspections
 


Test patch: