Repository: JuliaGPU/CUDA.jl

Issue #1053: Legacy cuIpc* APIs incompatible with stream-ordered allocator
Created: 2021-07-17T13:37:47Z
Using CUDA aware MPI with recent CUDA.jl generates
The call to cuIpcGetMemHandle failed. This means the GPU RDMA protocol
cannot be used.
  cuIpcGetMemHandle return value:   1


from the MPI layer.
The reason looks to be due to using cuMemCreate to manage memory in CUDA.jl is not consistent with using cuIpc* in OpenMPI on that memory.
According to this post https://developer.nvidia.com/blog/introducing-low-level-gpu-virtual-memory-management/ (see near end)
Operating system native interprocess communication

The new CUDA virtual memory management functions do not support the legacy cuIpc* functions with their memory. Instead, they expose a new mechanism for interprocess communication that works better with each supported platform. This new mechanism is based on manipulating system–specific handles. On Windows, these are of type HANDLE or D3DKMT_HANDLE, while on Linux-based platforms, these are file descriptors. 

To get one of these operating system–specific handles, the new function cuMemExportToShareableHandle is introduced. The appropriate request handle types must be passed to cuMemCreate. By default, memory is not exportable, so shareable handles are not available with the default properties.

After exporting the allocation to an OS-specific handle, you can transfer the handle to another process in the usual way: Linux can use Unix-domain sockets, Windows can use DuplicateHandle. The other process may then use cuMemImportFromShareableHandle and get back a CUmemGenericAllocationHandle value that the rest of the CUDA virtual memory management functions can work with.

The CUDA sample  shows how this works in practice. This sample works on all Linux and Windows platforms that support the CUDA virtual memory management functions.

On test systems the result is that communication fallback to non- GPU <-> GPU direct mechanisms. On AC922 IBM system the memory bandwidth drops from 90+GB/s to more in the region of 8GB/s etc.. and latencies appear to increase.
This "MWE" ( https://github.com/christophernhill/onan-jcon2021-bits/blob/00704dd28462a7688fcdb7484b2eaa560c1e869b/src/test_ca.jl#L1 ) will exhibit the behavior on a system with OpenMPI built with CUDA aware MPI enabled. The "MWE" also tries the using pool allocation kwargs ( https://github.com/christophernhill/onan-jcon2021-bits/blob/00704dd28462a7688fcdb7484b2eaa560c1e869b/src/test_ca.jl#L27 ) feature to see if that changes anything. The kwargs feature is not enough as the problem still exists.

Duplicate issue #1398: `cuIpcGetMemHandle` failure resulting in CUDA-aware MPI to fail
Created: 2022-02-23T19:57:00Z
Starting with CUDA v3.8.2, CUDA-aware MPI fails with following error, resulting in a segfault:
--------------------------------------------------------------------------
The call to cuIpcGetMemHandle failed. This means the GPU RDMA protocol
cannot be used.
  cuIpcGetMemHandle return value:   1
  address: 0xa15822400
Check the cuda.h file for what the return value means. Perhaps a reboot
of the node will clear the problem.
--------------------------------------------------------------------------

This failure was known in the past and required to set export JULIA_CUDA_MEMORY_POOL=none.
With CUDA v3.8.1, all works fine upon exporting export JULIA_CUDA_MEMORY_POOL=none. This suggests that a commit going from CUDA v3.8.1 to v3.8.2 introduced the bug. The (only) suspect may be #1383
To reproduce, you can run the ImplicitGlobalGrid test_update_halo.jl using MPI:
mpirun -np 2 julia ~/.julia/packages/ImplicitGlobalGrid/<hash>/test/test_update_halo.jl`

In the test one does not get a segfault, but warnings. In the application, it segfaults.
This occurs on Julia 1.7.1 and
julia> CUDA.versioninfo()
CUDA toolkit 11.4, local installation
NVIDIA driver 470.103.1, for CUDA 11.4
CUDA driver 11.6

Libraries: 
- CUBLAS: 11.6.5
- CURAND: 10.2.5
- CUFFT: 10.5.2
- CUSOLVER: 11.2.0
- CUSPARSE: 11.6.0
- CUPTI: 14.0.0
- NVML: 11.0.0+470.103.1
- CUDNN: missing
- CUTENSOR: missing

Toolchain:
- Julia: 1.7.1
- LLVM: 12.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

Environment:
- JULIA_CUDA_MEMORY_POOL: none
- JULIA_CUDA_USE_BINARYBUILDER: false

8 devices:
  0: NVIDIA A100-SXM4-40GB (sm_80, 39.583 GiB / 39.586 GiB available)
  1: NVIDIA A100-SXM4-40GB (sm_80, 39.583 GiB / 39.586 GiB available)
  2: NVIDIA A100-SXM4-40GB (sm_80, 39.583 GiB / 39.586 GiB available)
  3: NVIDIA A100-SXM4-40GB (sm_80, 39.583 GiB / 39.586 GiB available)
  4: NVIDIA A100-SXM4-40GB (sm_80, 39.583 GiB / 39.586 GiB available)
  5: NVIDIA A100-SXM4-40GB (sm_80, 39.583 GiB / 39.586 GiB available)
  6: NVIDIA A100-SXM4-40GB (sm_80, 39.583 GiB / 39.586 GiB available)
  7: NVIDIA A100-SXM4-40GB (sm_80, 39.583 GiB / 39.586 GiB available)

Closing comment (2022-02-23T20:16:44Z): Dup: #1053