Repository: spack/spack

Issue:
Installation issue: gromacs fail due to cuda compute_30 (dropped in cuda11)
Omitting a lot, as I believe nvcc from cuda-11.x simply will not generate compute_30 code, thus the error message.

See http://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

Default solution is to just remove compute_30.  More complex would be to allow different CUDA versions and adjust the generated code to match.

<!-- Thanks for taking the time to report this build failure. To proceed with the report please:

1. Title the issue "Installation issue: <name-of-the-package>".
2. Provide the information required below.

We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively! -->

### Steps to reproduce the issue

<!-- Fill in the exact spec you are trying to build and the relevant part of the error message -->
```console
$ spack install gromacs +cuda

     474    /gpfs/packages/spack/spack/opt/spack/linux-rhel7-broadwell/gcc-8.2.0/cuda-11.0.2-tlog2jv6ze6lfxk47omkfabtz7474gbw/bin/nvcc -M -D__CUDACC__ /tmp/swmgr/s
            pack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/gromacs/nbnxm/cuda/nbnxm_cuda.cu -o /tmp/swmgr/spack-stage/spack-s
            tage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-build/src/gromacs/CMakeFiles/libgromacs.dir/nbnxm/cuda/libgromacs_generated_nbnxm_cuda.cu.o.
            NVCC-depend -ccbin /gpfs/packages/spack/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/gcc-8.2.0-kot2sql3i2pckkfopvmxdmbdopuwy42t/bin/g++ -m64 -Dlibgroma
            cs_EXPORTS -DHAVE_CONFIG_H -DGMX_DOUBLE=0 -DUSE_STD_INTTYPES_H -Xcompiler ,\"-mavx2\",\"-mfma\",\"-pthread\",\"-fPIC\",\"-O2\",\"-g\",\"-DNDEBUG\",\"-f
            excess-precision=fast\",\"-funroll-all-loops\" -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm
            _37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -ge
            ncode arch=compute_70,code=sm_70 -gencode arch=compute_35,code=compute_35 -gencode arch=compute_50,code=compute_50 -gencode arch=compute_52,code=comput
            e_52 -gencode arch=compute_60,code=compute_60 -gencode arch=compute_61,code=compute_61 -gencode arch=compute_70,code=compute_70 -gencode arch=compute_7
            5,code=compute_75 -use_fast_math -DNVCC -I/gpfs/packages/spack/spack/opt/spack/linux-rhel7-broadwell/gcc-8.2.0/cuda-11.0.2-tlog2jv6ze6lfxk47omkfabtz747
            4gbw/include -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/external/thread_mpi/include -I/tmp/swmg
            r/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-build/src -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7
            untnzai2yrdnef7nklow2mc/spack-src/src -I/gpfs/packages/spack/spack/opt/spack/linux-rhel7-broadwell/gcc-8.2.0/openmpi-3.1.6-tg6m4mmn2p3vxbdpopsqgfesihlm
            3g5f/include -I/gpfs/packages/spack/spack/opt/spack/linux-rhel7-broadwell/gcc-8.2.0/hwloc-1.11.11-s54td4k4foxz3kdukoesox46risjltch/include -I/usr/inclu
            de -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/external -I/tmp/swmgr/spack-stage/spack-stage-gro
            macs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/external/tng_io/include -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai
            2yrdnef7nklow2mc/spack-build/tng/include -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/external/lm
            fit
     475    nvcc fatal   : Unsupported gpu architecture 'compute_30'


```

### Information on your system

<!-- Please include the output of `spack debug report` -->

<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- Please upload the following files. They should be present in the stage directory of the failing build. Also upload any config.log or similar file if one exists. -->
* [spack-build-out.txt]()
* [spack-build-env.txt]()

<!-- Some packages have maintainers who have volunteered to debug build failures. Run `spack maintainers <name-of-the-package>` and @mention them here if they exist. -->

### General information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [ ] I have run `spack maintainers <name-of-the-package>` and @mentioned any maintainers
- [ ] I have uploaded the build log and environment files
- [ ] I have searched the issues of this repo and believe this is not a duplicate



Discussion:
This seems to be coming from gromacs upstream:
```
./gromacs-2020.3/cmake/gmxManageNvccConfig.cmake:    list (APPEND GMX_CUDA_NVCC_GENCODE_FLAGS "-gencode;arch=compute_30,code=sm_30")
```

Looks like they haven't caught up to CUDA 11.  One solution would be to simply remove this line (via patch) if CUDA version is 11+.
@junghans @marvinbernhardt 
/cc @mabraham @pszi1ard @eirrgang 
Upstream issue at https://gitlab.com/gromacs/gromacs/-/issues/3632 
Changes proposed at https://gitlab.com/gromacs/gromacs/-/merge_requests/461 would likely work in the meantime (and similar would be needed for earlier GROMACS release branches, although those are no longer supported upstream).
 > Looks like they haven't caught up to CUDA 11.

Not yet, but on the other hand, CUDA 11 seems to have deprecated compute capability 3.0 overnight (?).

 > One solution would be to simply remove this line (via patch) if CUDA version is 11+.

For <=v2020.4 that is a reasonable workaround; alternatively, the spack config could explicitly override the CUDA targets on the CMake invocation, e.g. `-DGMX_CUDA_TARGET_SM="35;37;52;61;60;70;75" -DGMX_CUDA_TARGET_COMPUTE="70"` (could consider adding Ampere too, i.e. 80).

We have mostly K80s here (but a few newer cards).  Would be quite handy if this worked by default, or at least with some extra '+' flag or something.
> We have mostly K80s here (but a few newer cards). Would be quite handy if this worked by default, or at least with some extra '+' flag or something.

Not sure what do you mean by '+' flag, can you elaborate? As noted above you can pass those extra CMake flags at configure-time and you'll get essentially the same set of flags (and behavor) as the upstream fix implements.
I'm not exactly sure what I mean either, but I had in mind a "spack-ish" syntax to specify which cards to support, if that seems necessary. Just supporting all possible cards would be great as well, and maybe doesn't need a flag?

An additional possibility, though more work, would be to move all of this decision-making/flags to the 'cuda' spack module. Then spack packages that depend on that module could be taught to do what it said, rather than trying to guess the right values themselves.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, August 25, 2020 10:02 AM, Szilárd Páll <notifications@github.com> wrote:

>> We have mostly K80s here (but a few newer cards). Would be quite handy if this worked by default, or at least with some extra '+' flag or something.
>
> Not sure what do you mean by '+' flag, can you elaborate? As noted above you can pass those extra CMake flags at configure-time and you'll get essentially the same set of flags (and behavor) as the upstream fix implements.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, [view it on GitHub](https://github.com/spack/spack/issues/18239#issuecomment-680151106), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAABIAEASQSHH3Q3GBBBLTLSCPVDHANCNFSM4QJXJZEQ).
> I'm not exactly sure what I mean either, but I had in mind a "spack-ish" syntax to specify which cards to support, if that seems necessary. 

I can unfortunately not help with that.

> Just supporting all possible cards would be great as well, and maybe doesn't need a flag? 

The GROMACS build system already does that. In this case that mechanism does not work well enough because a GPU architecture was deprecated without advance notice and the GROMACS build system is not yet adapted to the CUDA 11 limitations.
> Changes proposed at https://gitlab.com/gromacs/gromacs/-/merge_requests/461 would likely work in the meantime (and similar would be needed for earlier GROMACS release branches, although those are no longer supported upstream).

Fix is merged and will be included in 2020.4
For a temporary solution: you can merely download their upstream release from [here](https://gitlab.com/gromacs/gromacs/-/archive/prepare-2020.4/gromacs-prepare-2020.4.tar.gz) that fixed the cuda-11 not compatible issue. and put them to `/path/to/spack/var/spack/cache/_source-cache/archive/cd/cd12bd4977e19533c1ea189f1c069913c86238a8a227c5a458140b865e6e3dc5.tar.gz`.

The checksum can be gotten by `sha256sum gromacs-prepare-2020.4.tar.gz`. Besides, you should modify the 2020.4 checksum in `/path/to/spack/var/spack/repos/builtin/packages/gromacs/package.py`, add ` version('2020.4', sha256='cd12bd4977e19533c1ea189f1c069913c86238a8a227c5a458140b865e6e3dc5')
`. 

Then you can simply run `spack install gromacs@2020.4+mpi+cuda`
@victoryang00, your link leads to a "404 Git Resource Not found".
2020.4 will be released soon, so let's just wait.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/gromacs/package.py b/var/spack/repos/builtin/packages/gromacs/package.py
--- a/var/spack/repos/builtin/packages/gromacs/package.py
+++ b/var/spack/repos/builtin/packages/gromacs/package.py
@@ -24,6 +24,7 @@ class Gromacs(CMakePackage):
     maintainers = ['junghans', 'marvinbernhardt']
 
     version('master', branch='master')
+    version('2020.4', sha256='5519690321b5500c7951aaf53ff624042c3edd1a5f5d6dd1f2d802a3ecdbf4e6')
     version('2020.3', sha256='903183691132db14e55b011305db4b6f4901cc4912d2c56c131edfef18cc92a9')
     version('2020.2', sha256='7465e4cd616359d84489d919ec9e4b1aaf51f0a4296e693c249e83411b7bd2f3')
     version('2020.1', sha256='e1666558831a3951c02b81000842223698016922806a8ce152e8f616e29899cf')


Test patch: