Repository: spack/spack

Issue:
Broken python and from_sourcing_file in packages depending indirectly on python
When a package *indirectly depends on python*, any attempt to run `EnvironmentModifications.from_sourcing_file` inside its installation environment ends up in an error. The `python` command is broken as well (and this is probably the real cause of the observed issue).

### Expected Result

To have that package installed.

### Actual Result

```console
$ spack install example-dependent
[lots of usual output...]
==> Executing phase: 'install'
==> Error: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
RuntimeError: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code

/home/spack/spack-repo-test/packages/example-dependent/package.py:16, in install:
     1             to_be_sourced = fs.join_path(prefix, 'SOURCE_ME')
     2             with open(to_be_sourced, 'w') as f:
     3                 f.write('export SOURCE_ME_HAS_BEEN_SOURCED=1')
  >> 4             EnvironmentModifications.from_sourcing_file(to_be_sourced).apply_modifications()
     5             fs.touch(fs.join_path(prefix, 'INSTALL_DONE'))

See build log for details:
  /home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.out
```

The actual error raised by the call is:

```console
$ cat /home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.out
==> Executing phase: 'install'
ImportError: No module named site
```

### Steps to reproduce the issue

I've managed to reduce the problem to a minimal set of packages that can be obtained [from GitHub](https://github.com/nazavode/spack-repo-test.git):

```console
$ git clone https://github.com/nazavode/spack-repo-test.git ~/spack-repo-test
$ spack repo add ~/spack-repo-test
```

That Spack repo contains two packages:

1. `example-dependency` that depends on `python`
2. `example-dependent` that depends on `example-dependency`

Having that repo added to Spack allows us to obtain this *interesting* spec:

```console
$ spack spec example-dependent
Input spec
--------------------------------
example-dependent

Concretized
--------------------------------
example-dependent@1.0.6%gcc@4.8.5 arch=linux-centos7-x86_64 
    ^example-dependency@1.0.6%gcc@4.8.5 arch=linux-centos7-x86_64 
        ^python@2.7.14%gcc@4.8.5 patches=123082ab3483ded78e86d7c809e98a804b3465b4683c96bd79a2fd799f572244 +pic+shared~tk~ucs4 arch=linux-centos7-x86_64 
            [a lot of other stuff python depends on...]
```

This minimal dependency graph is enough to trigger the issue:

```console
$ spack install example-dependent
[lots of usual output...]
==> Executing phase: 'install'
==> Error: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
RuntimeError: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code

/home/spack/spack-repo-test/packages/example-dependent/package.py:16, in install:
     1             to_be_sourced = fs.join_path(prefix, 'SOURCE_ME')
     2             with open(to_be_sourced, 'w') as f:
     3                 f.write('export SOURCE_ME_HAS_BEEN_SOURCED=1')
  >> 4             EnvironmentModifications.from_sourcing_file(to_be_sourced).apply_modifications()
     5             fs.touch(fs.join_path(prefix, 'INSTALL_DONE'))

See build log for details:
  /home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.out
```

The root of the problem lies in the fact that the `python` interpreter inside the installation environment is broken:

```console
$ spack install --only dependencies example-dependent
[installs just fine, output omitted...]
$ spack env example-dependent python
ImportError: No module named site
```

Inspecting the affected environment we get:

```console
$ spack env example-dependent bash
$ which python
/usr/bin/python
$ env | grep PYTHON
PYTHONHOME=/home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/python-2.7.14-4ipldugnhasdas2pjasdevq4ruha6biu
PYTHONPATH=
$ python
ImportError: No module named site
```

It looks like `^python@2.7.14` messed with our environment injecting that `PYTHONHOME` that our system's `/usr/bin/python` is not standing at all. The fun part is that *some* distro (Ubuntu 16.04 for instance) ships with a python that actually likes the *broken* environment and proved to be working fine (see below for details).

#### Wrap up

1. When a package depends **indirectly** on `python`, it ends up with:
    * `PYTHONHOME=<path to spack python prefix>`
    * `PYTHONPATH=`
    * `which python` is `<system python>`, which **is *broken* or not depending on its version**. This in turn breaks `EnvironmentModifications.from_sourcing_file` as well.
2. When a package depends **directly** on `python`, it ends up with:
    * `PYTHONHOME=<path to spack python prefix>`
    * `PYTHONPATH=<path to spack python prefix>`
    * `which python` is `<spack python>`, which works correctly.

### Information on your system

Reproduced this on a [centos7 docker image](https://gist.github.com/nazavode/87d073832894e71a7cf12db06e6b6aed) that ships with `python 2.7.5` while tested to be **working fine** on a [ubuntu 16.04 docker image](https://gist.github.com/nazavode/50c6336f73fca09cf0c4b0997747df08) that ships with `python 2.7.12`. Spack was cd084d02ce56a076a21b61bb1c4850dec74e2b4d.
A `Dockefile` that executes the whole issue reproduction procedure can be found [here](https://gist.github.com/nazavode/61650268b150e94bdbfa58fe0f626373).



Discussion:
That `PYTHONHOME` variable is messing around, see #6965 

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -511,7 +511,12 @@ def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
         """Set PYTHONPATH to include the site-packages directory for the
         extension and any other python extensions it depends on."""
 
+        # If we set PYTHONHOME, we must also ensure that the corresponding
+        # python is found in the build environment. This to prevent cases
+        # where a system provided python is run against the standard libraries
+        # of a Spack built python. See issue #7128
         spack_env.set('PYTHONHOME', self.home)
+        spack_env.prepend_path('PATH', os.path.dirname(self.command.path))
 
         python_paths = []
         for d in dependent_spec.traverse(


Test patch: