Repository: spack/spack

Issue:
Can't install flang or gcc variants with CUDA support
I am a complete beginner to using spack.

Both flang and gcc support OpenMP offloading - this is the future of supercomputing - just one version of your code with OpenMP offloading to support any accelerator device, GPU, Vector Engine, etc.

I have not had any problems installing gcc or flang,  with CUDA support, using install scripts dowloaded from the web.

However, with spack, I have had problems in both cases:
[pgmath.txt](https://github.com/spack/spack/files/5277018/pgmath.txt)
[gcc_cuda.txt](https://github.com/spack/spack/files/5277019/gcc_cuda.txt)
[flang_cuda.txt](https://github.com/spack/spack/files/5277021/flang_cuda.txt)




When I try to install flang, the installation errors out with pgmath install.

When I try to install flang +cuda, the installation does not even begin due to 'conflict' diagnostics.

I can install gcc, but gcc +nvptx errors out.

* **Spack:** 0.15.4-1058-84dbf69
* **Python:** 3.6.8
* **Platform:** linux-centos7-skylake_avx512


Discussion:
@michaelkuhn @naromero77 @trws @ax3l @svenevs 
I can take a quick look later today, but one thing to note is that the OpenMP 4.5 offload in flang is very incomplete. You can use it to learn OpenMP 4.5 with small test cases, but a real OpenMP 4.5 application is out of the question.

There are several problems here. Let me start with the conflicts obtain with `spack install flang+cuda`. GCC 10 is simply to new and will not work with CUDA. Try an older GCC -- I use GCC 7.x most of the time, it looks like GCC 9 should work. It may also be that the CUDA Spack package needs to be update to support GCC 10.x.

You are hitting a bug in pgmath that is fixed in `master`, please update your local Spack installation. Or do,
`spack install flang@master%gcc`
with your current installation.

I don't maintain GCC, but I took a quick look any way. It was not obvious what is wrong from your log file.
4. "%gcc@8:" conflicts with "llvm-flang+cuda ^cuda@:10.0.130 arch=linux-None-x86_64:"
5. "%gcc@9:" conflicts with "llvm-flang+cuda ^cuda@:10.2.89 arch=linux-None-x86_64:"
6. "%gcc@:4,10:" conflicts with "llvm-flang+cuda ^cuda@:11.0.2 arch=linux-None-x86_64:"

Your build depends on more than one version of CUDA, which cannot be done. Not sure what fix is, flang has never been functional for me. It usually doesn't update quick enough for newer CUDA FYI.

Your spec also needs to set cuda_arch to a non-none value e.g. 50 if you are compiling for sm50 arch (depends on what you are targeting, we can't detect or assume e.g. consider node building without any GPUs that dispatches to either sm50 or sm60 depending on compute node) so e.g. cuda_arch=50,60 on spack spec.

AFAIU openmp isn't really related here, at least in terms of getting it to build. You want to build in CUDA support, for which one mechanism of accessing that support is through openmp. Just trying to clarify priority, with CUDA support the openmp side building on that (implemented in compiler) should just work :tm:
You are correct. There is a double-dependency here on CUDA that is un-needed.

Also, part of the problem with this implementation is that `cuda_arch` is not automatically propagated properly in the DAG. The value of `cuda_arch` is set to none, because I find it equally problematic to set it to some default value. You have to compile the bitcode for your GPU architecture, normally this is not such a big issue except if cuda_arch value is not supported by your version of CUDA.

I think its possible to remediate a bit, but I am also thinking of just getting rid of the OpenMP offload from classic flang. It is incomplete implementation of the OpenMP 4.5 standard and NVidia is putting all the energy into their new Flang that is LLVM -- not classic flang.
> except if cuda_arch value is not supported by your version of CUDA.

That's the only thing actually being guaranteed / enforced out of this btw. Different packages / build systems may decide to do their own thing regardless. But yeah it would be better if cuda_arch were somewhere higher than package level. My understanding is when compilers are "properly in the DAG" then we can move cuda_arch there. Gets a little tricky with llvm+cuda (?) Maybe it just gets moved to `config.yaml` and the nvcc or clang++ or flang (or pgi or gcc...) compilers grab it from there rather than needing to specify for both nvcc and clang++ and flang?  AKA no concrete plans I know of / ideas welcome, I'm not up to speed on any of the new concretizer stuff.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/flang/package.py b/var/spack/repos/builtin/packages/flang/package.py
--- a/var/spack/repos/builtin/packages/flang/package.py
+++ b/var/spack/repos/builtin/packages/flang/package.py
@@ -8,7 +8,7 @@
 import os
 
 
-class Flang(CMakePackage, CudaPackage):
+class Flang(CMakePackage):
     """Flang is a Fortran compiler targeting LLVM."""
 
     homepage = "https://github.com/flang-compiler/flang"
@@ -24,17 +24,27 @@ class Flang(CMakePackage, CudaPackage):
     version('20180921', sha256='f33bd1f054e474f1e8a204bb6f78d42f8f6ecf7a894fdddc3999f7c272350784')
     version('20180612', sha256='6af858bea013548e091371a97726ac784edbd4ff876222575eaae48a3c2920ed')
 
+    # variants
+    msg = ('Target OpenMP offloading to NVIDIA GPUs '
+           '(experimental w/ no support). '
+           'Only works with GCC compilers compatible with CUDA 9.x. '
+           'Specify ^llvm-flang+cuda cuda_arch=<value> explictly.')
+
+    variant('nvptx',
+            default=False,
+            description=msg)
+
     # Patched only relevant for March 2019 release with OpenMP Offload support
     patch('https://github.com/flang-compiler/flang/commit/b342225a64692d2b9c3aff7658a8e4f94a8923eb.diff',
           sha256='3bd2c7453131eaaf11328785a3031fa2298bdd0c02cfd5e2b478e6e847d5da43',
-          when='@20190329 +cuda')
+          when='@20190329 +nvptx')
 
     # Build dependency
     depends_on('cmake@3.8:', type='build')
     depends_on('python@2.7:', type='build')
 
-    depends_on('llvm-flang@release_90', when='@master')
-    depends_on('llvm-flang@20190329', when='@20190329')
+    depends_on('llvm-flang@release_90', when='@master ~nvptx')
+    depends_on('llvm-flang@20190329', when='@20190329 ~nvptx')
     depends_on('llvm-flang@20181226_70', when='@20181226')
     depends_on('llvm-flang@20180921', when='@20180921')
     depends_on('llvm-flang@20180612', when='@20180612')
@@ -45,10 +55,11 @@ class Flang(CMakePackage, CudaPackage):
     depends_on('pgmath@20180921', when='@20180921')
     depends_on('pgmath@20180612', when='@20180612')
 
-    depends_on('llvm-flang +cuda', when='+cuda')
+    depends_on('llvm-flang+cuda@release_70', when='@master +nvptx')
+    depends_on('llvm-flang+cuda@20190329', when='@20190329 +nvptx')
 
     # conflicts
-    conflicts('+cuda', when='@:20181226',
+    conflicts('+nvptx', when='@:20181226',
               msg='OpenMP offload to NVidia GPUs available 20190329 or later')
 
     # Spurious problems running in parallel the Makefile
diff --git a/var/spack/repos/builtin/packages/llvm-flang/package.py b/var/spack/repos/builtin/packages/llvm-flang/package.py
--- a/var/spack/repos/builtin/packages/llvm-flang/package.py
+++ b/var/spack/repos/builtin/packages/llvm-flang/package.py
@@ -16,7 +16,7 @@ class LlvmFlang(CMakePackage, CudaPackage):
 
     maintainer = ['naromero77']
 
-    version('release_90', branch='release_90')
+    version('release_90', branch='release_90', preferred=True)
     version('release_80', branch='release_80')
     version('release_70', branch='release_70')
     version('release_60', branch='release_60')
@@ -46,7 +46,11 @@ class LlvmFlang(CMakePackage, CudaPackage):
     # libomptarget dependencies
     depends_on('libelf', when='+cuda')
     depends_on('libffi', when='+cuda')
-    depends_on('cuda@:9', when='+cuda')  # llvm 7 not compatible with newer version of cuda
+    # note that there may be other llvm-flang + cuda conflicts, but these
+    # are the only versions that are relevant for the flang+nvptx package
+    # llvm 7 not compatible with newer version of cuda
+    depends_on('cuda@:9', when='+cuda@release_70')
+    depends_on('cuda@:9', when='+cuda@20190329')
 
     # LLVM-Flang Componentes: Driver, OpenMP
     resource(name='flang-driver',


Test patch: