Repository: cupy/cupy
Pull request: Add `NCCL_ERROR_REMOTE_ERROR` to the set of errors from NCCL
Error codes possible from nccl - https://github.com/NVIDIA/nccl/blob/2ea4ee94bfb04c886c79ccae60ac9961000fdee2/src/nccl.h.in#L43

Fixes/Returns a more reasonable error for #8661 

Problem statement:
NcclCommunicator initialization fails with KeyError: 6
### Description

NCCL communicator initialization fails with -

```
comm = nccl.NcclCommunicator(comm_size, self.nccl_comm_id, comm_rank)
  File "cupy_backends/cuda/libs/nccl.pyx", line 282, in cupy_backends.cuda.libs.nccl.NcclCommunicator.__init__
  File "cupy_backends/cuda/libs/nccl.pyx", line 128, in cupy_backends.cuda.libs.nccl.check_status
  File "cupy_backends/cuda/libs/nccl.pyx", line 117, in cupy_backends.cuda.libs.nccl.NcclError.__init__
KeyError: 6
```

NOTE: We dont' see this error consistently and sometimes restarting the process fixes it.

### To Reproduce

```py
# Write the code here
```


### Installation

Wheel (`pip install cupy-***`)

### Environment

```
OS                           : Linux-6.1.85+-x86_64-with-glibc2.35
Python Version               : 3.10.12
CuPy Version                 : 13.3.0
CuPy Platform                : NVIDIA CUDA
NumPy Version                : 1.26.1
SciPy Version                : 1.12.0
Cython Build Version         : 0.29.36
Cython Runtime Version       : None
CUDA Root                    : /usr/local/cuda
nvcc PATH                    : /usr/local/cuda/bin/nvcc
CUDA Build Version           : 12060
CUDA Driver Version          : 12040
CUDA Runtime Version         : 12060 (linked to CuPy) / 12030 (locally installed)
CUDA Extra Include Dirs      : []
cuBLAS Version               : (available)
cuFFT Version                : 11012
cuRAND Version               : 10304
cuSOLVER Version             : (11, 5, 4)
cuSPARSE Version             : (available)
NVRTC Version                : (12, 3)
Thrust Version               : 200600
CUB Build Version            : 200600
Jitify Build Version         : <unknown>
cuDNN Build Version          : 8801
cuDNN Version                : 8906
NCCL Build Version           : 21602
NCCL Runtime Version         : 22105
cuTENSOR Version             : None
cuSPARSELt Build Version     : None
Device 0 Name                : NVIDIA H100 80GB HBM3
Device 0 Compute Capability  : 90
Device 0 PCI Bus ID          : 0000:04:00.0
Device 1 Name                : NVIDIA H100 80GB HBM3
Device 1 Compute Capability  : 90
Device 1 PCI Bus ID          : 0000:05:00.0
Device 2 Name                : NVIDIA H100 80GB HBM3
Device 2 Compute Capability  : 90
Device 2 PCI Bus ID          : 0000:0B:00.0
Device 3 Name                : NVIDIA H100 80GB HBM3
Device 3 Compute Capability  : 90
Device 3 PCI Bus ID          : 0000:0C:00.0
Device 4 Name                : NVIDIA H100 80GB HBM3
Device 4 Compute Capability  : 90
Device 4 PCI Bus ID          : 0000:84:00.0
Device 5 Name                : NVIDIA H100 80GB HBM3
Device 5 Compute Capability  : 90
Device 5 PCI Bus ID          : 0000:85:00.0
Device 6 Name                : NVIDIA H100 80GB HBM3
Device 6 Compute Capability  : 90
Device 6 PCI Bus ID          : 0000:8B:00.0
Device 7 Name                : NVIDIA H100 80GB HBM3
Device 7 Compute Capability  : 90
Device 7 PCI Bus ID          : 0000:8C:00.0
```


### Additional Information

_No response_

Hints from issue discussion:

Commit message: cupy: add remote error to the set of errors from nccl

Patch to review:
diff --git a/cupy_backends/cuda/libs/nccl.pyx b/cupy_backends/cuda/libs/nccl.pyx
index 1ac2c4ee006..3e1607f13fd 100644
--- a/cupy_backends/cuda/libs/nccl.pyx
+++ b/cupy_backends/cuda/libs/nccl.pyx
@@ -98,6 +98,7 @@ cdef dict ERROR2 = {
     3: 'NCCL_ERROR_INTERNAL_ERROR',
     4: 'NCCL_ERROR_INVALID_ARGUMENT',
     5: 'NCCL_ERROR_INVALID_USAGE',
+    6: 'NCCL_ERROR_REMOTE_ERROR'
 }
 
 


Code review comments:
diff_hunk: @@ -98,6 +98,7 @@ cdef dict ERROR2 = {
     3: 'NCCL_ERROR_INTERNAL_ERROR',
     4: 'NCCL_ERROR_INVALID_ARGUMENT',
     5: 'NCCL_ERROR_INVALID_USAGE',
+    6: 'NCCL_ERROR_REMOTE_ERROR'
path: cupy_backends/cuda/libs/nccl.pyx
text: @user1:
```suggestion
    6: 'NCCL_ERROR_REMOTE_ERROR',
```