Repository: spack/spack

Issue:
nlcglib+cuda does not pass `cuda_arch` to `kokkos` dependency correctly
### Steps to reproduce

If I request `cuda_arch=70`, this is not passed on correctly to `kokkos`, which gets concretized with `cuda_arch=60` (the default I guess?):

```console
spack spec -U -I nlcglib%gcc+cuda cuda_arch=70
Input spec
--------------------------------
 -   nlcglib%gcc+cuda cuda_arch=70

Concretized
--------------------------------
 -   nlcglib@1.0b%gcc@11.3.0+cuda~ipo+openmp~rocm~tests build_system=cmake build_type=Release cuda_arch=70 generator=make arch=linux-almalinux8-x86_64_v3
[+]      ^cmake@3.26.3%gcc@11.3.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-almalinux8-x86_64_v3
[+]          ^curl@8.1.2%gcc@11.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-almalinux8-x86_64_v3
[+]              ^nghttp2@1.52.0%gcc@11.3.0 build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]              ^openssl@3.1.2%gcc@11.3.0~docs+shared build_system=generic certs=mozilla arch=linux-almalinux8-x86_64_v3
[+]                  ^ca-certificates-mozilla@2023-05-30%gcc@11.3.0 build_system=generic arch=linux-almalinux8-x86_64_v3
[+]              ^pkgconf@1.9.5%gcc@11.3.0 build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]          ^ncurses@6.4%gcc@11.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]          ^zlib@1.2.13%gcc@11.3.0+optimize+pic+shared build_system=makefile arch=linux-almalinux8-x86_64_v3
[+]      ^cuda@11.8.0%gcc@11.3.0~allow-unsupported-compilers~dev build_system=generic arch=linux-almalinux8-x86_64_v3
[+]          ^libxml2@2.10.3%gcc@11.3.0~python build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]              ^libiconv@1.17%gcc@11.3.0 build_system=autotools libs=shared,static arch=linux-almalinux8-x86_64_v3
[+]              ^xz@5.4.1%gcc@11.3.0~pic build_system=autotools libs=shared,static arch=linux-almalinux8-x86_64_v3
[+]      ^gmake@4.4.1%gcc@11.3.0~guile build_system=autotools arch=linux-almalinux8-x86_64_v3
[e]      ^hpcx-mpi@2.15%gcc@11.3.0 build_system=generic arch=linux-almalinux8-x86_64_v3
 -       ^kokkos@4.1.00%gcc@11.3.0~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr+cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning+wrapper build_system=cmake build_type=Release cuda_arch=60 cxxstd=17 generator=make intel_gpu_arch=none arch=linux-almalinux8-x86_64_v3
 -           ^kokkos-nvcc-wrapper@4.0.01%gcc@11.3.0 build_system=generic arch=linux-almalinux8-x86_64_v3
 -       ^nlohmann-json@3.11.2%gcc@11.3.0~ipo+multiple_headers build_system=cmake build_type=Release generator=make arch=linux-almalinux8-x86_64_v3
 -       ^openblas@0.3.23%gcc@11.3.0~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-almalinux8-x86_64_v3
[+]          ^perl@5.38.0%gcc@11.3.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-almalinux8-x86_64_v3
[+]              ^berkeley-db@18.1.40%gcc@11.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-almalinux8-x86_64_v3
[+]              ^bzip2@1.0.8%gcc@11.3.0~debug~pic+shared build_system=generic arch=linux-almalinux8-x86_64_v3
[+]                  ^diffutils@3.9%gcc@11.3.0 build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]              ^gdbm@1.23%gcc@11.3.0 build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]                  ^readline@8.2%gcc@11.3.0 build_system=autotools patches=bbf97f1 arch=linux-almalinux8-x86_64_v3
```

Moreover, if I request **multiple** GPU architectures (e.g. `cuda_arch=70,80`), I still get kokkos with `cuda_arch=60`.
(By the way, kokkos does not support multiple GPU architectures (see #38722 , fixed by #38723), so it should eventually throw an error in this case).
```console
$ spack spec -U -I nlcglib%gcc+cuda cuda_arch=70,80
Input spec
--------------------------------
 -   nlcglib%gcc+cuda cuda_arch=70,80

Concretized
--------------------------------
 -   nlcglib@1.0b%gcc@11.3.0+cuda~ipo+openmp~rocm~tests build_system=cmake build_type=Release cuda_arch=70,80 generator=make arch=linux-almalinux8-x86_64_v3
[+]      ^cmake@3.26.3%gcc@11.3.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-almalinux8-x86_64_v3
[+]          ^curl@8.1.2%gcc@11.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-almalinux8-x86_64_v3
[+]              ^nghttp2@1.52.0%gcc@11.3.0 build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]              ^openssl@3.1.2%gcc@11.3.0~docs+shared build_system=generic certs=mozilla arch=linux-almalinux8-x86_64_v3
[+]                  ^ca-certificates-mozilla@2023-05-30%gcc@11.3.0 build_system=generic arch=linux-almalinux8-x86_64_v3
[+]              ^pkgconf@1.9.5%gcc@11.3.0 build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]          ^ncurses@6.4%gcc@11.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]          ^zlib@1.2.13%gcc@11.3.0+optimize+pic+shared build_system=makefile arch=linux-almalinux8-x86_64_v3
[+]      ^cuda@11.8.0%gcc@11.3.0~allow-unsupported-compilers~dev build_system=generic arch=linux-almalinux8-x86_64_v3
[+]          ^libxml2@2.10.3%gcc@11.3.0~python build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]              ^libiconv@1.17%gcc@11.3.0 build_system=autotools libs=shared,static arch=linux-almalinux8-x86_64_v3
[+]              ^xz@5.4.1%gcc@11.3.0~pic build_system=autotools libs=shared,static arch=linux-almalinux8-x86_64_v3
[+]      ^gmake@4.4.1%gcc@11.3.0~guile build_system=autotools arch=linux-almalinux8-x86_64_v3
[e]      ^hpcx-mpi@2.15%gcc@11.3.0 build_system=generic arch=linux-almalinux8-x86_64_v3
 -       ^kokkos@4.1.00%gcc@11.3.0~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr+cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning+wrapper build_system=cmake build_type=Release cuda_arch=60 cxxstd=17 generator=make intel_gpu_arch=none arch=linux-almalinux8-x86_64_v3
 -           ^kokkos-nvcc-wrapper@4.0.01%gcc@11.3.0 build_system=generic arch=linux-almalinux8-x86_64_v3
 -       ^nlohmann-json@3.11.2%gcc@11.3.0~ipo+multiple_headers build_system=cmake build_type=Release generator=make arch=linux-almalinux8-x86_64_v3
 -       ^openblas@0.3.23%gcc@11.3.0~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-almalinux8-x86_64_v3
[+]          ^perl@5.38.0%gcc@11.3.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-almalinux8-x86_64_v3
[+]              ^berkeley-db@18.1.40%gcc@11.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-almalinux8-x86_64_v3
[+]              ^bzip2@1.0.8%gcc@11.3.0~debug~pic+shared build_system=generic arch=linux-almalinux8-x86_64_v3
[+]                  ^diffutils@3.9%gcc@11.3.0 build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]              ^gdbm@1.23%gcc@11.3.0 build_system=autotools arch=linux-almalinux8-x86_64_v3
[+]                  ^readline@8.2%gcc@11.3.0 build_system=autotools patches=bbf97f1 arch=linux-almalinux8-x86_64_v3
```


### Error message

_No response_

### Information on your system

* **Spack:** 0.21.0.dev0
* **Python:** 3.6.8
* **Platform:** linux-centos7-cascadelake
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/nlcglib/package.py b/var/spack/repos/builtin/packages/nlcglib/package.py
--- a/var/spack/repos/builtin/packages/nlcglib/package.py
+++ b/var/spack/repos/builtin/packages/nlcglib/package.py
@@ -49,9 +49,12 @@ class Nlcglib(CMakePackage, CudaPackage, ROCmPackage):
         depends_on("rocblas")
         depends_on("rocsolver")
 
-    with when("+cuda"):
-        depends_on("kokkos+cuda+cuda_lambda+wrapper", when="%gcc")
-        depends_on("kokkos+cuda")
+    for arch in CudaPackage.cuda_arch_values:
+        depends_on(
+            f"kokkos+cuda+cuda_lambda+wrapper cuda_arch={arch}",
+            when=f"%gcc +cuda cuda_arch={arch}",
+        )
+        depends_on(f"kokkos+cuda cuda_arch={arch}", when=f"+cuda cuda_arch={arch}")
 
     def cmake_args(self):
         options = [


Test patch: