Repository: cupy/cupy

Issue:
Workaround ROCm 4.3 `LLVM_PATH` issue in hipRTC
Seems ROCm 4.3.1 still have this issue

https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html#hiprtc-file-not-found-error

> Suggested Workarounds
> ...
> Add “-I <path to ROCm>/llvm/lib/clang/13.0.0/include/” to compiler options in the call to hiprtcCompileProgram (). Note, this workaround requires the following changes in the code:




Discussion:
Here is a reproducer. It seems the problem only occurs when multi-versioned install is used?

(Sorry for the dirty Dockerfile...)

```dockerfile
FROM ubuntu:20.04

RUN apt update && apt -y install curl ca-certificates
RUN apt -y install gnupg2
RUN echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/debian/ ubuntu main' | tee /etc/apt/sources.list.d/rocm.list
RUN curl -L https://repo.radeon.com/rocm/rocm.gpg.key | apt-key add -
RUN apt update && apt -y install hipblas4.3.0 hipsparse4.3.0 rocsparse4.3.0 rocrand4.3.0 rocthrust4.3.0 rocsolver4.3.0 rocfft4.3.0 hipcub4.3.0 rocprim4.3.0 rccl4.3.0
RUN apt -y install python3-pip
RUN apt -y install initramfs-tools
RUN apt -y install rock-dkms
RUN export DEBIAN_FRONTEND=noninteractive && apt -y install rocm-dev4.3.0
RUN apt -y install libnuma-dev
RUN apt -y install hipfft4.3.0
RUN pip install 'cupy-rocm-4-3==10.0.0b3' -f https://pip.cupy.dev/pre
ENV LD_LIBRARY_PATH /opt/rocm-4.3.0/lib
```

```
docker build -t reproducer .
docker run --device=/dev/kfd --device=/dev/dri --rm -it reproducer
```

```
root@19709ad8d1a7:/# python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cupy
>>> cupy.arange(10)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/cupy/cuda/compiler.py", line 650, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 136, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 148, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends/cuda/libs/nvrtc.pyx", line 67, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: HIPRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/cupy/_creation/ranges.py", line 57, in arange
    _arange_ufunc(typ(start), typ(step), ret, dtype=dtype)
  File "cupy/_core/_kernel.pyx", line 1185, in cupy._core._kernel.ufunc.__call__
  File "cupy/_core/_kernel.pyx", line 1212, in cupy._core._kernel.ufunc._get_ufunc_kernel
  File "cupy/_core/_kernel.pyx", line 952, in cupy._core._kernel._get_ufunc_kernel
  File "cupy/_core/_kernel.pyx", line 62, in cupy._core._kernel._get_simple_elementwise_kernel
  File "cupy/_core/core.pyx", line 2078, in cupy._core.core.compile_with_cache
  File "/usr/local/lib/python3.8/dist-packages/cupy/cuda/compiler.py", line 453, in compile_with_cache
    return _compile_with_cache_hip(
  File "/usr/local/lib/python3.8/dist-packages/cupy/cuda/compiler.py", line 816, in _compile_with_cache_hip
    base = _preprocess_hiprtc('', options)
  File "/usr/local/lib/python3.8/dist-packages/cupy/cuda/compiler.py", line 739, in _preprocess_hiprtc
    result, _ = prog.compile(options)
  File "/usr/local/lib/python3.8/dist-packages/cupy/cuda/compiler.py", line 667, in compile
    raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: <built-in>:1:10: fatal error: '__clang_hip_runtime_wrapper.h' file not found
#include "__clang_hip_runtime_wrapper.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated when compiling for gfx900.
Error: Failed to compile opencl source (from CL or HIP source to LLVM IR).
```
<!-- Added by Auto Notify Bot (hip) -->
cc/ @amathews-amd

Resolving patch:
diff --git a/.pfnci/generate.py b/.pfnci/generate.py
--- a/.pfnci/generate.py
+++ b/.pfnci/generate.py
@@ -122,11 +122,6 @@ def generate_dockerfile(self) -> str:
                 'ENV LD_LIBRARY_PATH "${ROCM_HOME}/lib"',
                 'ENV CPATH "${ROCM_HOME}/include"',
                 'ENV LDFLAGS "-L${ROCM_HOME}/lib"',
-
-                # In ROCm 4.3, hiprtc has a problem that it can not find a
-                # header file related to LLVM/clang. As a workaround, we
-                # temporarily give LLVM_PATH here. See #5592.
-                'ENV LLVM_PATH "${ROCM_HOME}/llvm"',
                 '',
             ]
 
diff --git a/cupy/cuda/compiler.py b/cupy/cuda/compiler.py
--- a/cupy/cuda/compiler.py
+++ b/cupy/cuda/compiler.py
@@ -11,6 +11,7 @@
 
 from cupy.cuda import device
 from cupy.cuda import function
+from cupy.cuda import get_rocm_path
 from cupy_backends.cuda.api import driver
 from cupy_backends.cuda.api import runtime
 from cupy_backends.cuda.libs import nvrtc
@@ -808,6 +809,12 @@ def _compile_with_cache_hip(source, options, arch, cache_dir, extra_source,
     #   ROCm-Developer-Tools/HIP#2248
     options += ('-fcuda-flush-denormals-to-zero',)
 
+    # Workaround ROCm 4.3 LLVM_PATH issue in hipRTC #5689
+    rocm_build_version = driver.get_build_version()
+    if rocm_build_version >= 40300000 and rocm_build_version < 40500000:
+        options += (
+            '-I' + get_rocm_path() + '/llvm/lib/clang/13.0.0/include/',)
+
     if cache_dir is None:
         cache_dir = get_cache_dir()
     # As of ROCm 3.5.0 hiprtc/hipcc can automatically pick up the


Test patch:
diff --git a/.pfnci/linux/tests/rocm-4-0.Dockerfile b/.pfnci/linux/tests/rocm-4-0.Dockerfile
--- a/.pfnci/linux/tests/rocm-4-0.Dockerfile
+++ b/.pfnci/linux/tests/rocm-4-0.Dockerfile
@@ -24,7 +24,6 @@ ENV ROCM_HOME "/opt/rocm"
 ENV LD_LIBRARY_PATH "${ROCM_HOME}/lib"
 ENV CPATH "${ROCM_HOME}/include"
 ENV LDFLAGS "-L${ROCM_HOME}/lib"
-ENV LLVM_PATH "${ROCM_HOME}/llvm"
 
 RUN git clone https://github.com/pyenv/pyenv.git /opt/pyenv
 ENV PYENV_ROOT "/opt/pyenv"
diff --git a/.pfnci/linux/tests/rocm-4-2.Dockerfile b/.pfnci/linux/tests/rocm-4-2.Dockerfile
--- a/.pfnci/linux/tests/rocm-4-2.Dockerfile
+++ b/.pfnci/linux/tests/rocm-4-2.Dockerfile
@@ -24,7 +24,6 @@ ENV ROCM_HOME "/opt/rocm"
 ENV LD_LIBRARY_PATH "${ROCM_HOME}/lib"
 ENV CPATH "${ROCM_HOME}/include"
 ENV LDFLAGS "-L${ROCM_HOME}/lib"
-ENV LLVM_PATH "${ROCM_HOME}/llvm"
 
 RUN git clone https://github.com/pyenv/pyenv.git /opt/pyenv
 ENV PYENV_ROOT "/opt/pyenv"
diff --git a/.pfnci/linux/tests/rocm-4-3.Dockerfile b/.pfnci/linux/tests/rocm-4-3.Dockerfile
--- a/.pfnci/linux/tests/rocm-4-3.Dockerfile
+++ b/.pfnci/linux/tests/rocm-4-3.Dockerfile
@@ -24,7 +24,6 @@ ENV ROCM_HOME "/opt/rocm"
 ENV LD_LIBRARY_PATH "${ROCM_HOME}/lib"
 ENV CPATH "${ROCM_HOME}/include"
 ENV LDFLAGS "-L${ROCM_HOME}/lib"
-ENV LLVM_PATH "${ROCM_HOME}/llvm"
 
 RUN git clone https://github.com/pyenv/pyenv.git /opt/pyenv
 ENV PYENV_ROOT "/opt/pyenv"