Repository: spack/spack

Issue:
Installation issue: spark+hadoop
### The problem

When spark is built with hadoop, the comand `hadoop classpath` is run to setup the environment:

spark/package.py
```python
    @when('+hadoop')
    def setup_environment(self, spack_env, run_env):
        hadoop = self.spec['hadoop'].command
        hadoop_classpath = hadoop('classpath', output=str)

        # Remove whitespaces, as they can compromise syntax in
        # module files
        hadoop_classpath = re.sub('[\s+]', '', hadoop_classpath)

        run_env.set('SPARK_DIST_CLASSPATH', hadoop_classpath)
```

However, this command needs the `JAVA_HOME` variable set, which does not happen either 

- before `spark.setup_environment` is called xor
- in the `hadoop` subprocess

causing the build to fail when a java jdk is not already in the environment (and may not be the same JDK that the rest of hadoop and spark will be built with).

### Possible solution

Put the `JAVA_HOME` into the `hadoop classpath` subprocess environment:

```diff
 diff --git a/var/spack/repos/builtin/packages/spark/package.py b/var/spack/repos/builtin/packages/spark/package.py
index 9dce9b81f..aa50245c8 100644
--- a/var/spack/repos/builtin/packages/spark/package.py
+++ b/var/spack/repos/builtin/packages/spark/package.py
@@ -68,6 +68,7 @@ class Spark(Package):
     @when('+hadoop')
     def setup_environment(self, spack_env, run_env):
         hadoop = self.spec['hadoop'].command
+        hadoop.add_default_env('JAVA_HOME', self.spec['java'].home)
         hadoop_classpath = hadoop('classpath', output=str)
 
         # Remove whitespaces, as they can compromise syntax in

```

### Steps to reproduce the issue

```console
$ ./bin/spack spec -lINt 'spark@2.3.0%gcc@6.2.0+hadoop'                                                                                                                                                                     
Input spec                                                                                                                                                                                                                                     
--------------------------------                                                                                                                                                                                                               
     [    ]  spark@2.3.0%gcc@6.2.0+hadoop                                                                                                                                                                                                      
                                                                                                                                                                                                                                               
Concretized                                                                                                                                                                                                                                    
--------------------------------                                                                                                                                                                                                               
     pxqqqhv  [    ]  builtin.spark@2.3.0%gcc@6.2.0+hadoop arch=linux-rhel7-x86_64                                                                                                                                                             
[+]  572r7i7  [b r ]      ^builtin.hadoop@3.1.1%gcc@6.2.0 arch=linux-rhel7-x86_64                                                                                                                                                              
[+]  vrwoesl  [b r ]          ^builtin.jdk@10.0.2_13%gcc@6.2.0 arch=linux-rhel7-x86_64                                                                                                                                                         
                                                                                                                                                                                                                                               
$ ./bin/spack spec -lINt 'spark@2.3.0%gcc@6.2.0+hadoop ^jdk@10.0.2_13'                                                                                                                                                      
Input spec                                                                                                                                                                                                                                     
--------------------------------                                                                                                                                                                                                               
     [    ]  spark@2.3.0%gcc@6.2.0+hadoop                                                                                                                                                                                                      
     [    ]      ^jdk@10.0.2_13                                                                                                                                                                                                                
                                                                                                                                                                                                                                               
Concretized                                                                                                                                                                                                                                    
--------------------------------                                                                                                                                                                                                               
     pxqqqhv  [    ]  builtin.spark@2.3.0%gcc@6.2.0+hadoop arch=linux-rhel7-x86_64                                                                                                                                                             
[+]  572r7i7  [b r ]      ^builtin.hadoop@3.1.1%gcc@6.2.0 arch=linux-rhel7-x86_64                                                                                                                                                              
[+]  vrwoesl  [b r ]          ^builtin.jdk@10.0.2_13%gcc@6.2.0 arch=linux-rhel7-x86_64                                                                                                                                                         
                                                                                                                                                                                                                                               
$ ./bin/spack install -v 'spark@2.3.0%gcc@6.2.0+hadoop ^jdk@10.0.2_13'                                                                                                                                                      
==> jdk is already installed in /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/opt/spack/20180914/linux-rhel7-x86_64/gcc-6.2.0/jdk-10.0.2_13-vrwoesl2avpqbyben7qhedo7jwflaxit                                                                 
==> hadoop is already installed in /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/opt/spack/20180914/linux-rhel7-x86_64/gcc-6.2.0/hadoop-3.1.1-572r7i7vahxzfjuowa26xpxicuv6xytq                                                               
==> Installing spark                                                                                                                                                                                                                           
ERROR: JAVA_HOME is not set and could not be found.                                                                                                                                                                                            
==> Error: ProcessError: Command exited with status 1:                                                                                                                                                                                         
    '/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/opt/spack/20180914/linux-rhel7-x86_64/gcc-6.2.0/hadoop-3.1.1-572r7i7vahxzfjuowa26xpxicuv6xytq/bin/hadoop' 'classpath'
$ ./bin/spack env 'spark@2.3.0%gcc@6.2.0+hadoop ^jdk@10.0.2_13'                          
ERROR: JAVA_HOME is not set and could not be found.
==> Error: Command exited with status 1:                                                    
'/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/opt/spack/20180914/linux-rhel7-x86_64/gcc-6.2.0/hadoop-3.1.1-572r7i7vahxzfjuowa26xpxicuv6xytq/bin/hadoop' 'classpath'
```

### Platform and user environment

Please report your OS here:
```commandline
$ uname -a                                                                                                                                                                                                                 
Linux rhea-login8g 3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux 
$ lsb_release -d                                                                                                                                                                                                           
Description:    Red Hat Enterprise Linux Server release 7.5 (Maipo)
``` 

### Additional information

Error with stack trace:
```console
$ ./bin/spack -d env 'spark@2.3.0%gcc@6.2.0+hadoop ^jdk@10.0.2_13'                                                                    
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/defaults/modules.yaml
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/defaults/linux/modules.yaml
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/modules.yaml
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/defaults/config.yaml
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/config.yaml
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/defaults/repos.yaml
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/repos.yaml
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/defaults/packages.yaml
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/packages.yaml
==> READ LOCK: /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/var/spack/mcache/providers/.builtin-index.yaml.lock[0:0] [Acquiring]
==> READ LOCK: /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/var/spack/mcache/providers/.builtin-index.yaml.lock[0:0] [Acquired]
==> READ LOCK: /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/var/spack/mcache/providers/.builtin-index.yaml.lock[0:0] [Released]
==> READ LOCK: /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/var/spack/mcache/providers/.olcf-index.yaml.lock[0:0] [Acquiring]
==> READ LOCK: /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/var/spack/mcache/providers/.olcf-index.yaml.lock[0:0] [Acquired]
==> READ LOCK: /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/var/spack/mcache/providers/.olcf-index.yaml.lock[0:0] [Released]
==> Reading config file /autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/etc/spack/compilers.yaml
==> Reading config file /ccs/home/uamntr/.spack/linux/compilers.yaml
==> DATABASE LOCK TIMEOUT: 120s
==> PACKAGE LOCK TIMEOUT: No timeout
==> '/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/opt/spack/20180914/linux-rhel7-x86_64/gcc-6.2.0/hadoop-3.1.1-572r7i7vahxzfjuowa26xpxicuv6xytq/bin/hadoop' 'classpath'
ERROR: JAVA_HOME is not set and could not be found.
==> Error: Command exited with status 1:
'/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/opt/spack/20180914/linux-rhel7-x86_64/gcc-6.2.0/hadoop-3.1.1-572r7i7vahxzfjuowa26xpxicuv6xytq/bin/hadoop' 'classpath'
Traceback (most recent call last):
  File "/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/lib/spack/spack/main.py", line 653, in main
    return _invoke_command(command, parser, args, unknown)
  File "/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/lib/spack/spack/main.py", line 432, in _invoke_command
    return_val = command(parser, args)
  File "/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/lib/spack/spack/cmd/env.py", line 69, in env
    build_env.setup_package(spec.package, args.dirty)
  File "/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/lib/spack/spack/build_environment.py", line 666, in setup_package
    pkg.setup_environment(spack_env, run_env)
  File "/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/lib/spack/spack/multimethod.py", line 127, in __call__
    return method(package_self, *args, **kwargs)
  File "/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/var/spack/repos/builtin/packages/spark/package.py", line 72, in setup_environment

  File "/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/lib/spack/spack/util/executable.py", line 208, in __call__
    proc.returncode, long_msg)
spack.util.executable.ProcessError: Command exited with status 1:
    '/autofs/nccs-svm1_sw/.staging/rhea/.swci/0-core/opt/spack/20180914/linux-rhel7-x86_64/gcc-6.2.0/hadoop-3.1.1-572r7i7vahxzfjuowa26xpxicuv6xytq/bin/hadoop' 'classpath'
```




Discussion:

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/spark/package.py b/var/spack/repos/builtin/packages/spark/package.py
--- a/var/spack/repos/builtin/packages/spark/package.py
+++ b/var/spack/repos/builtin/packages/spark/package.py
@@ -68,6 +68,7 @@ def install_dir(dirname):
     @when('+hadoop')
     def setup_environment(self, spack_env, run_env):
         hadoop = self.spec['hadoop'].command
+        hadoop.add_default_env('JAVA_HOME', self.spec['java'].home)
         hadoop_classpath = hadoop('classpath', output=str)
 
         # Remove whitespaces, as they can compromise syntax in


Test patch: