Repository: spack/spack

Issue:
Installation issue: yaksa@0.2 +cuda %gcc@12.3.0
### Steps to reproduce the issue

I ran into this when trying to build `mpich %gcc@12.3.0 +cuda netmod=ucx pmi=pmix ^cuda@11.8 +allow-unsupported-compilers`

```console
$ spack spec -l mpich%gcc@12.3.0+cuda netmod=ucx pmi=pmix ^cuda@11.8
Input spec
--------------------------------
 -   mpich%gcc@12.3.0+cuda netmod=ucx pmi=pmix
 -       ^cuda@11.8
 
 Concretized
--------------------------------
 -   vl25zhp  mpich@4.1.2%gcc@12.3.0~argobots+cuda+fortran~hcoll+hwloc~hydra+libxml2+pci~rocm+romio~slurm~vci~verbs+wrapperrpath build_system=autotools cuda_arch=80 datatype-engine=auto device=ch4 netmod=ucx pmi=pmix arch=linux-rhel8-zen2
[e]  tatyw2o      ^cuda@11.8%gcc@12.3.0+allow-unsupported-compilers~dev build_system=generic arch=linux-rhel8-zen2
[+]  oaezgck      ^findutils@4.9.0%gcc@12.3.0 build_system=autotools patches=440b954 arch=linux-rhel8-zen2
[+]  wda76ac      ^hwloc@2.9.1%gcc@12.3.0~cairo+cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools cuda_arch=80 libs=shared,static arch=linux-rhel8-zen2
[+]  v3xl3ra          ^ncurses@6.4%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel8-zen2
[e]  jxaxqp2          ^pkgconf@1.4.2%gcc@12.3.0 build_system=autotools arch=linux-rhel8-zen2
[+]  a25ibkd      ^libpciaccess@0.17%gcc@12.3.0 build_system=autotools arch=linux-rhel8-zen2
[e]  ljdxawj          ^libtool@2.4.6%gcc@12.3.0 build_system=autotools arch=linux-rhel8-zen2
[-]  6gqawar          ^util-macros@1.19.3%gcc@12.3.0 build_system=autotools arch=linux-rhel8-zen2
[+]  6tqy7tj      ^libxml2@2.10.3%gcc@12.3.0+pic~python+shared build_system=autotools arch=linux-rhel8-zen2
[+]  gmdjnou          ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-rhel8-zen2
[+]  wv6bhnf          ^xz@5.4.1%gcc@12.3.0+pic build_system=autotools libs=shared,static arch=linux-rhel8-zen2
[+]  gl4jfjp          ^zlib-ng@2.1.3%gcc@12.3.0+compat+opt build_system=autotools patches=299b958,ae9077a,b692621 arch=linux-rhel8-zen2
[e]  jxaxqp2      ^pkgconf@1.4.2%gcc@12.3.0 build_system=autotools arch=linux-rhel8-zen2
[+]  iiz3ymd      ^pmix@5.0.1%gcc@12.3.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-rhel8-zen2
[+]  zpd4eva          ^libevent@2.1.12%gcc@12.3.0+openssl build_system=autotools arch=linux-rhel8-zen2
[+]  vfrxdbw              ^openssl@3.1.3%gcc@12.3.0~docs+shared build_system=generic certs=mozilla arch=linux-rhel8-zen2
[-]  f7zqyjr                  ^ca-certificates-mozilla@2023-05-30%gcc@12.3.0 build_system=generic arch=linux-rhel8-zen2
[e]  mbywbad                  ^perl@5.26.3%gcc@12.3.0+cpanm+opcode+open+shared+threads build_system=generic patches=8cf4302 arch=linux-rhel8-zen2
[+]  giw5sed      ^ucx@1.14.1%gcc@12.3.0~assertions~backtrace_detail~cma~cuda~dc~debug~dm+examples~gdrcopy~gtest~ib_hw_tm~java~knem~logging~mlx5_dv+numa+openmp+optimizations~parameter_checking+pic~rc~rdmacm~rocm+thread_multiple~ucg~ud~verbs~vfs~xpmem build_system=autotools libs=shared,static opt=3 simd=auto arch=linux-rhel8-zen2
[+]  2hrbodu          ^numactl@2.0.14%gcc@12.3.0 build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-rhel8-zen2
[e]  34ciftm              ^autoconf@2.69%gcc@12.3.0 build_system=autotools patches=7793209 arch=linux-rhel8-zen2
[e]  4awjffi              ^automake@1.16.1%gcc@12.3.0 build_system=autotools arch=linux-rhel8-zen2
[e]  v6g426c              ^m4@1.4.18%gcc@12.3.0+sigsegv build_system=autotools patches=3877ab5,fc9b616 arch=linux-rhel8-zen2
 -   3gp6itg      ^yaksa@0.2%gcc@12.3.0+cuda~rocm build_system=autotools cuda_arch=80 arch=linux-rhel8-zen2
[e]  34ciftm          ^autoconf@2.69%gcc@12.3.0 build_system=autotools patches=7793209 arch=linux-rhel8-zen2
[e]  4awjffi          ^automake@1.16.1%gcc@12.3.0 build_system=autotools arch=linux-rhel8-zen2
[e]  ljdxawj          ^libtool@2.4.6%gcc@12.3.0 build_system=autotools arch=linux-rhel8-zen2
[e]  v6g426c          ^m4@1.4.18%gcc@12.3.0+sigsegv build_system=autotools patches=3877ab5,fc9b616 arch=linux-rhel8-zen2
[e]  vij5hij          ^python@3.11.4%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-rhel8-zen2
```



### Error message

<details><summary>Error message</summary>

<pre>
==> Error: ProcessError: Command exited with status 1:
    '/scratch/wp11/spack/stage/spack-stage-yaksa-0.2-3gp6itg5vlljusyvsifmuuceq7q77vp5/spack-src/configure' '--prefix=/projects/deploy/install/linux-rhel8-zen2/gcc-12.3.0/yaksa-0.2-3gp6itg5vlljusyvsifmuuceq7q77vp5' '--with-cuda=/usr/local/cuda-11.8' '--with-cuda-sm=80' '--without-hip'

7 errors found in build log:
     28     libtoolize: copying file 'm4/ltsugar.m4'
     29     libtoolize: copying file 'm4/ltversion.m4'
     30     libtoolize: copying file 'm4/lt~obsolete.m4'
     31     autoreconf: running: /projects/pkgs/autotools-ac-2.69-am-1.15.1-lt-2.4.6/bin/autoconf --force
     32     autoreconf: running: /projects/pkgs/autotools-ac-2.69-am-1.15.1-lt-2.4.6/bin/autoheader --force
     33     autoreconf: running: automake --add-missing --copy --force-missing
  >> 34     configure.ac:81: installing 'm4/ar-lib'
  >> 35     configure.ac:78: installing 'm4/compile'
  >> 36     configure.ac:83: installing 'm4/config.guess'
  >> 37     configure.ac:83: installing 'm4/config.sub'
  >> 38     configure.ac:74: installing 'm4/install-sh'
  >> 39     configure.ac:74: installing 'm4/missing'
     40     Makefile.am: installing 'm4/depcomp'
     41     autoreconf: Leaving directory `.'
     42     === done ===
     43     
     44     Generating a helper maint/Version... done
     45     ==> yaksa: Executing phase: 'configure'

     ...

     198    checking alignment of long double... 16
     199    checking cuda_runtime_api.h usability... yes
     200    checking cuda_runtime_api.h presence... yes
     201    checking for cuda_runtime_api.h... yes
     202    checking for cudaStreamSynchronize in -lcudart... yes
     203    checking whether nvcc works... no
  >> 204    configure: error: CUDA was requested but it is not functional
</pre></details>


### Information on your system

* **Spack:** 0.21.0.dev0 (b6f08f1d4ef3b0ca56430c1a465d660828b45ed2)
* **Python:** 3.6.8
* **Platform:** linux-rhel8-zen2
* **Concretizer:** clingo

### Additional information

- [spack-build-out.txt](https://github.com/spack/spack/files/12778111/spack-build-out.txt)
- [spack-build-env.txt](https://github.com/spack/spack/files/12778112/spack-build-env.txt)

I can confirm that the `nvcc` from CUDA 11.8 in the system works just fine by compiling and running a CUDA dot product test code.

```
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
```

Summoning `yaksa` maintainers @hzhou @raffenet @yfguo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
The issue is `nvcc` erroring out because of the incompatible host compiler. I'm not sure how `+allow-unsupported-compilers` works. Yaksa supports setting `NVCC_FLAGS=-allow-unsupported-compiler`, but maybe there is another convention we should follow?
@raffenet Ah I see. Looks like `NVCC_FLAGS` needs to be added to the `yaksa` Spack recipe? Any of these can be used as a template:
```
var/spack/repos/builtin/packages/aluminum/package.py:179:            if spec.satisfies("%cce") and spec.satisfies("^cuda+allow-unsupported-compilers"):
var/spack/repos/builtin/packages/dihydrogen/package.py:179:            if spec.satisfies("%cce") and spec.satisfies("^cuda+allow-unsupported-compilers"):
var/spack/repos/builtin/packages/hydrogen/package.py:189:            if spec.satisfies("%cce") and spec.satisfies("^cuda+allow-unsupported-compilers"):
var/spack/repos/builtin/packages/lbann/package.py:411:            if spec.satisfies("%cce") and spec.satisfies("^cuda+allow-unsupported-compilers"):
var/spack/repos/builtin/packages/nlcglib/package.py:91:        if "^cuda+allow-unsupported-compilers" in self.spec:
```

For instance:
https://github.com/spack/spack/blob/75ed26258ccbf84b9bc6dd11d8a9ccadbd54ff9e/var/spack/repos/builtin/packages/hydrogen/package.py#L189-L190

On the other hand, without injecting the `NVCC_FLAGS` when building `yaksa`, that means I need to stick to GCC >=12 for CUDA 12.

https://github.com/spack/spack/blob/75ed26258ccbf84b9bc6dd11d8a9ccadbd54ff9e/lib/spack/spack/build_systems/cuda.py#L138

However, I can't Spack-build `gcc@12.3.0 +nvptx ^cuda@12.2` yet... See https://github.com/spack/spack/issues/36172 for details. Perhaps I need to start a PR to fix that issue... I think it just needs bumping the minimum PTX arch to `sm_53` for CUDA 12, since that CUDA version dropped support for `sm_30` and `sm_35`.
> @raffenet Ah I see. Looks like `NVCC_FLAGS` needs to be added to the `yaksa` Spack recipe? Any of these can be used as a template:

Thanks, I'll try to open up a PR later today.

> However, I can't Spack-build `gcc@12.3.0 +nvptx ^cuda@12.2` yet... See #36172 for details. Perhaps I need to start a PR to fix that issue... I think it just needs bumping the minimum PTX arch to `sm_53` for CUDA 12, since that CUDA version dropped support for `sm_30` and `sm_35`.

We just made some updates to the sm settings that should also significantly cut down on build time and library size. It's probably time we tag another release, so I'll try to tackle both at the same time.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/yaksa/package.py b/var/spack/repos/builtin/packages/yaksa/package.py
--- a/var/spack/repos/builtin/packages/yaksa/package.py
+++ b/var/spack/repos/builtin/packages/yaksa/package.py
@@ -47,6 +47,8 @@ def configure_args(self):
             cuda_archs = spec.variants["cuda_arch"].value
             if "none" not in cuda_archs:
                 config_args.append("--with-cuda-sm={0}".format(",".join(cuda_archs)))
+            if "^cuda+allow-unsupported-compilers" in self.spec:
+                config_args.append("NVCC_FLAGS=-allow-unsupported-compiler")
 
         if "+rocm" in spec:
             config_args.append("--with-hip={0}".format(spec["hip"].prefix))


Test patch: