Repository: cupy/cupy

Issue:
Separate compilation not working on Windows
The following code runs fine on Linux but fails on Windows:

```py
import cupy
kernel = cupy.RawKernel('extern "C" __global__ void test() {}', 'test', backend='nvcc', enable_cooperative_groups=True)
kernel.compile()
```

Error raised in Windows:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cupy\core\raw.pyx", line 282, in cupy.core.raw.RawKernel.compile
  File "cupy\core\raw.pyx", line 110, in cupy.core.raw.RawKernel._kernel
  File "cupy\cuda\function.pyx", line 234, in cupy.cuda.function.Module.get_function
  File "cupy\cuda\function.pyx", line 175, in cupy.cuda.function.Function.__init__
  File "cupy_backends\cuda\api\driver.pyx", line 262, in cupy_backends.cuda.api.driver.moduleGetFunction
  File "cupy_backends\cuda\api\driver.pyx", line 124, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_NOT_FOUND: named symbol not found
```

Internally, the above code does the following nvcc separate compilation:

```
nvcc -gencode=arch=compute_70,code=sm_70 -ftz=true -rdc=true -Xcompiler -fPIC -shared -o test_rdc.o test.cu
nvcc -gencode=arch=compute_70,code=sm_70 --cubin -ftz=true -rdc=true -Xcompiler -fPIC -shared test_rdc.o --device-link -o test_rdc.cubin
```

It generates a correct cubin on Linux:

```
% cuobjdump -symbols test_rdc.cubin

symbols:
STT_FUNC         STB_GLOBAL STO_ENTRY      test
```

But on Windows it generates an empty (?) cubin, which is causing `named symbol not found` error:

```
C:\Users\chainer\Documents>cuobjdump -all -symbols test_rdc.cubin

symbols:
```

It seems we need some other commands to do separate compilation on Windows.

cc/ @leofang @anaruse


Discussion:
This might be a dumb question, but why do we need a separate compilation?
Two examples are cooperative group or dynamic parallelism, where a device linking (to the static device runtime library) must happen, and so a separate compilation is required.
@kmaehashi I found CMake provides an option `CUDA_RESOLVE_DEVICE_SYMBOLS` to address this, but I don't know which compiler flag(s) it translates to: https://stackoverflow.com/a/51353324/2344149
I'm not familiar with separate compilation, but what happens if you compile with the following options?

    nvcc -gencode=arch=compute_70,code=sm_70 -v --device-c -ftz=true test.cu -o test.o 
    nvcc -gencode=arch=compute_70,code=sm_70 -v --device-link --cubin -ftz=true test.o -o test.cubin

I think it's what we do?
I don't know well about Windows, but the Windows compiler probably dosen't have a -fPIC option, does it?

    nvcc -gencode=arch=compute_70,code=sm_70 -ftz=true -rdc=true -Xcompiler -fPIC -shared -o test_rdc.o test.cu
    nvcc -gencode=arch=compute_70,code=sm_70 --cubin -ftz=true -rdc=true -Xcompiler -fPIC -shared test_rdc.o --device-link -o test_rdc.cubin


Interesting...Yes for compiling the CUDA codes at build time, I do notice that MSVC does *not* add `-fPIC` to `postargs`:
https://github.com/cupy/cupy/blob/fa3a87494068d46816f7305735cfa3661cc568b3/cupy_setup_build.py#L1014
So perhaps we could give it a shot? Though if it's the problem, nvcc should have raised an error...


Resolving patch:
diff --git a/cupy/cuda/compiler.py b/cupy/cuda/compiler.py
--- a/cupy/cuda/compiler.py
+++ b/cupy/cuda/compiler.py
@@ -448,10 +448,8 @@ def _compile_with_cache_cuda(
     options += ('-ftz=true',)
 
     if enable_cooperative_groups:
-        # `cooperative_groups` requires `-rdc=true`.
-        # The three latter flags are to resolve linker error.
-        # (https://devtalk.nvidia.com/default/topic/1023604/linker-error/)
-        options += ('-rdc=true', '-Xcompiler', '-fPIC', '-shared')
+        # `cooperative_groups` requires relocatable device code.
+        options += ('--device-c',)
 
     if _get_bool_env_variable('CUPY_CUDA_COMPILE_WITH_DEBUG', False):
         options += ('--device-debug', '--generate-line-info')


Test patch: