Repository: spack/spack

Issue:
making HIP and ROCM more robust
The HIP and ROCM packages for AMD GPU's have a lot of dependencies and
often do not build cleanly.  I'd like to use this issue for a
discussion of problems and solutions for making `spack install hip`
work more robustly.

----------

(1) Here's where I'm currently stuck at `spack install comgr`.

This is on x86_64, RH 8.2, gcc 8.3.1, spack at commit 676d68a979 on
Nov 18, just after 0.16.0 and just before a couple of troublesome
commits for ROCM.  The RH HIP/ROCM packages for 3.8.0 are installed at
/opt/rocm.  But I'm also trying to build in spack from scratch.

```
% spack install -v comgr @3.8.0
...
Call Stack (most recent call first):
  /home/krentel/rocm/160/install/linux-rhel8-x86_64/gcc-8.3.1/cmake-3.15.7-papjvxann3gzrdnc75na4
  x7qk7gv3lq7/share/cmake-3.15/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /opt/rocm/hip/lib/cmake/hip/hip-config.cmake:112 (find_dependency)
  test/CMakeLists.txt:122 (find_package)
...
1 error found in build log:
  >> 27    CMake Error at /home/krentel/rocm/160/spack-repo/build-stage/spack-stage-comgr-3.8.0-2celuqx77r52he
           uhiub4awesu4iln567/spack-build-2celuqx/lib/cmake/amd_comgr/amd_comgr-config.cmake:3 (include):
     28      The file
     29    
     30        /home/krentel/rocm/160/spack-repo/build-stage/spack-stage-comgr-3.8.0-2celuqx77r52heuhiub4awesu
           4iln567/spack-build-2celuqx/lib/cmake/amd_comgr/amd_comgr-targets.cmake
     31    
     32      was generated by the export() command.  It may not be used as the argument
     33      to the include() command.  Use ALIAS targets instead to refer to targets by
```

The error is something in cmake about improperly using the output of
export() as the argument for include().  But note that the call stack
includes a cmake file from /opt/rocm/.  I have a full HIP install at
/opt/rocm, but nothing in spack ever refers to it, so this should not
happen.

And indeed, if I move /opt/rocm/ out of the way, then the build
succeeds.  So, the problem is that the ROCM cmake files are looking
into /opt/rocm/ when spack is trying to build everything from scratch.

This is probably something that needs to be fixed at the ROCM level,
outside of spack.  Theoretically, spack could add patches, but it
really needs to be fixed in ROCM/comgr.

I suspect there is a general problem that the ROCM cmake files are
looking into /opt/rocm/ and /usr when they shouldn't.  For example,
this is what the patches in llvm-amdgpu are doing for ncurses and
zlib, fixing cmake to prefer the spack builds over /usr.

Anyway, this is problem 1, where I'm stuck now.

@srekolam @haampie You want to add your war stories or solutions?




Discussion:
I usually open a PR on the ROCm package, and then add a patch to spack. The problem is that upstream PRs remain open for a long time and sometimes do not get merged.

In this case, you want to make sure that comgr does not build the tests:

https://github.com/RadeonOpenCompute/ROCm-CompilerSupport/blob/amd-stg-open/lib/comgr/test/CMakeLists.txt#L123

which apparently looks for an installation of hip although comgr is in our a case a dependency of hip itself :upside_down_face: 

The LLVM issues you're pointing out are really issues with LLVM itself. They should be fixed in LLVM 12 at least.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/comgr/package.py b/var/spack/repos/builtin/packages/comgr/package.py
--- a/var/spack/repos/builtin/packages/comgr/package.py
+++ b/var/spack/repos/builtin/packages/comgr/package.py
@@ -24,13 +24,20 @@ class Comgr(CMakePackage):
 
     variant('build_type', default='Release', values=("Release", "Debug"), description='CMake build type')
 
+    # Disable the hip compile tests.  Spack should not be using
+    # /opt/rocm, and this breaks the build when /opt/rocm exists.
+    patch('hip-tests.patch')
+
+    depends_on('cmake@3.2.0:',  type='build', when='@:3.8.99')
+    depends_on('cmake@3.13.4:', type='build', when='@3.9.0:')
+
     depends_on('zlib', type='link')
     depends_on('z3', type='link')
     depends_on('ncurses', type='link')
-    depends_on('cmake@3:', type='build')
+
     for ver in ['3.5.0', '3.7.0', '3.8.0', '3.9.0', '3.10.0']:
         depends_on('llvm-amdgpu@' + ver, type='build', when='@' + ver)
         depends_on('rocm-device-libs@' + ver, type='build', when='@' + ver)
         depends_on('rocm-cmake@' + ver, type='build', when='@' + ver)
 
-    root_cmakelists_dir = 'lib/comgr'
+    root_cmakelists_dir = join_path('lib', 'comgr')


Test patch: