Repository: spack/spack

Issue:
Cray environment for CUDAHOSTCXX is goofed
<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `spack find` to list all the installed packages and ..." -->

### Steps to reproduce the issue

On CoriGPU I am building lbann using GCC 8.2.0 and cmake is unable to resolve CUDAHOSTCXX because spack setups up a cray environment even though I am not using one.  It ends up with CUDAHOSTCXX=CC, which is wrong.


### Error Message

<!-- If Spack reported an error, provide the error message. If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->

```console
CMake Error at /global/u2/v/vanessen/spack.git/opt/spack/cray-cnl7-skylake_avx512/gcc-8.2.0/cmake-3.18.0-esuryqzyhasr24bju42ydqhlnvlvmd4b/share/cmake-3.18/Modules/CMakeDetermineCUDACompiler.cmake:46 (message):
  Could not find compiler set in environment variable CUDAHOSTCXX:
  CC.
Call Stack (most recent call first):
  CMakeLists.txt:264 (enable_language)
```

```console
[lbann-dev-skylake_avx512] vanessen@cgpu12:~/lbann.git/build/gnu.Release.cgpu.nersc.gov/build> printenv CUDAHOSTCXX
CC
```

### Information on your system

<!-- Please include the output of `spack debug report` -->

<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

CoriGPU

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
@scheibelp @tgamblin @becker33 

Why does the the compiler on a cray system get set to "CC" rather than the actual compiler.  In the cuda package this cause problems with the following line:

```console
env.set('CUDAHOSTCXX', self.compiler.cxx) 
```

This should be set to the actual compiler the same way it is done on other systems.
Notes:

* `CUDAHOSTCXX` is set in https://github.com/spack/spack/pull/16869
* @bvanessen is not referring to builds with Spack, but the fact that once the package is installed, that this variable is set in their environment (affecting subsequent installs outside of Spack)
@robertmaynard @ax3l @becker33: #16869 seems to break things on Cray because we set the CUDAHOSTCXX to `CC`, as that is what we detect things as in `compilers.yaml`.

When `nvcc` is called on Cray does it ignore some module settings or something? Seems like if we use `CC` it should pick up the modules loaded in the build environment and do the right thing.

If that doesn't work, what should we be setting `CUDAHOSTCXX` to there?  It should really be the cxx the spec concretized with.


Ok we discussed this a bit and it's... interesting.  I think the fundamental problem in @ax3l's #16869 is this, from `cuda/package.py`:

```python
    def setup_run_environment(self, env):
        env.set('CUDA_HOME', self.prefix)
        env.set('CUDAHOSTCXX', self.compiler.cxx)    # <-- this line
```

This says that we should set `CUDAHOSTCXX` in the *run* environment for CUDA.  But that is assuming that the run environment has something to do with the way CUDA was built.  In @bvanessen's case, he's trying to load an environment that somehow depends on CUDA.  But envs can have multiple specs in them and they may not all be built with the same compiler as CUDA.  So `self.compiler.cxx` is wrong -- that is CUDA's compiler.  The run env shouldn't be steered towards it.

In a *spack* build, we could use `setup_dependent_build_environment()` (@ax3l: was #16869 supposed to set that instead of `setup_build_environment()`?), but I suspect we'd want to do something different there too -- we should point `CUDAHOSTCXX` at the `CXX` for the *dependent*, not for CUDA.  The dependent is what is being built, and we know its compiler.

I have no real issue setting `CUDAHOSTCXX` in the CUDA build environment, but does that even do anything?  I think that part of #16869 might be wrong, too, if that was the intent.

I think the fix here is to remove `env.set('CUDAHOSTCXX', self.compiler.cxx)` from the run environment and let users of environments (and users of apps depending on CUDA) set their own `CUDAHOSTCXX`.

Thoughts?

@adamjstewart fYI
Uff, I am sorry for the regression.

Yes, I actually want this for dependent packages that build against CUDA with CMake - so `setup_dependent_build_environment`... And I would like that var set when activating an environment with CUDA in it or `spack load`-ing CUDA, because that's how I would use it for compiling things against Spack envs/modules in development.

If this conflicts badly with other workflows please just revert it. I am just starting a two week PTO :)
> es, I actually want this for dependent packages that build against CUDA with CMake - so setup_dependent_build_environment

This, I think, is fine.  But test it on Cori.  Can you do that and submit a PR?

> I would like that var set when activating an environment with CUDA in it or spack load-ing CUDA, because that's how I would use it for compiling things against Spack envs/modules in development.

This is the tricky part.  We don't currently have a good way to do that because the env doesn't really have a "compiler" associated with it.  You load specs in the env but we don't know what you intend to build with it.  So I would just remove this for now.

I do think this is a good thing for a dev env to set but it's part of a larger discussion around dev workflow.  We don't currently really have a way for devs to say what they *intend* to build in the environment.  Only int he spack build environment do we know that (which is why you could do it in `setup_dependent_build_environment()`).

Kind of curious how @robo-wylder sets things like this.
> > I actually want this for dependent packages that build against CUDA with CMake - so setup_dependent_build_environment
>
> This, I think, is fine. But test it on Cori. Can you do that and submit a PR?

Sorry, don't have time for this right now:
> I am just starting a two week PTO :)

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/cuda/package.py b/var/spack/repos/builtin/packages/cuda/package.py
--- a/var/spack/repos/builtin/packages/cuda/package.py
+++ b/var/spack/repos/builtin/packages/cuda/package.py
@@ -80,15 +80,16 @@ class Cuda(Package):
     depends_on('libxml2', when='@10.1.243:')
 
     def setup_build_environment(self, env):
-        env.set('CUDAHOSTCXX', self.compiler.cxx)
         if self.spec.satisfies('@10.1.243:'):
             libxml2_home  = self.spec['libxml2'].prefix
             env.set('LIBXML2HOME', libxml2_home)
             env.append_path('LD_LIBRARY_PATH', libxml2_home.lib)
 
+    def setup_dependent_build_environment(self, env, dependent_spec):
+        env.set('CUDAHOSTCXX', dependent_spec.package.compiler.cxx)
+
     def setup_run_environment(self, env):
         env.set('CUDA_HOME', self.prefix)
-        env.set('CUDAHOSTCXX', self.compiler.cxx)
 
     def install(self, spec, prefix):
         if os.path.exists('/tmp/cuda-installer.log'):


Test patch: