Repository: pytorch/pytorch

Issue:
Handling the error message of cudaGetDeviceCount
Summary of the suggestion: The correct behavior of `is_available()` is it should return False whenever `cudaGetDeviceCount(&count) != cudaSuccess || count == 0`, also for the check in `start_threads`

Details:
This issue is strongly related to #1154, but I think is should be more appropriate as a new issue. A snapshot of this issue is the pytorch crashes when the `cudaGetDeviceCount` throw some unknown exceptions. This unknown behavior happens when the cuda or device is improperly installed. For example, the author of that issue, get things fixed by uninstalling an old version of cuda.

But I'm wondering whether Pytorch should handle this by backoff to CPU whenever it happens. The current code looks inconsistent for handling this. For example, in https://github.com/pytorch/pytorch/blob/master/torch/cuda/__init__.py#L24-L34
It catches two types of exceptions, the `cudaErrorInsufficientDriver` and `cudaErrorNoDevice`, which seems a little bit limited. (A side note: catching `cudaErrorNoDevice` by string matching at https://github.com/pytorch/pytorch/blob/master/torch/cuda/__init__.py#L32 looks a little bit hacky to me :-) ) 

I think the correct behavior of `is_available()` is it should return False whenever `cudaGetDeviceCount(&count) != cudaSuccess || count == 0`. This will make the code more robust to the unexpected error exception throw by some incorrectly configure CUDA. For my own case, the the CUDA returns `cudaErrorUnknown` that crashes `is_available()`. I asked my admins to fix, but they think it is extensive and unnecessary "because the CPU nodes don't support gpu jobs, the particular errors that they throw are irrelevant."

The same thing happens at (as far as I can tell) https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/engine.cpp#L353-L361 as well, where it only checks `cudaErrorInsufficientDriver`. (somehow it doesn't crashes on `cudaErrorNoDevice`, for which I don't know why). I think it also should be a no-success-then-false behavior as I suggested in `is_available()`. A less important questions is why the cuda check happens here? If I've called `is_available()` before then call `.backward()`, Pytorch seems detect the device twice. Which looks unnecessary to me. This flag should be set when launching the pytorch I guess.


Discussion:

Resolving patch:
diff --git a/torch/cuda/__init__.py b/torch/cuda/__init__.py
--- a/torch/cuda/__init__.py
+++ b/torch/cuda/__init__.py
@@ -26,12 +26,7 @@ def is_available():
     if (not hasattr(torch._C, '_cuda_isDriverSufficient') or
             not torch._C._cuda_isDriverSufficient()):
         return False
-    try:
-        return torch._C._cuda_getDeviceCount() > 0
-    except RuntimeError as e:
-        if 'no CUDA-capable device is detected' in e.args[0]:
-            return False
-        raise
+    return torch._C._cuda_getDeviceCount() > 0
 
 
 def _sleep(cycles):
@@ -87,8 +82,8 @@ def _lazy_init():
         raise RuntimeError(
             "Cannot re-initialize CUDA in forked subprocess. " + msg)
     _check_driver()
-    assert torch._C._cuda_init()
-    assert torch._C._cuda_sparse_init()
+    torch._C._cuda_init()
+    torch._C._cuda_sparse_init()
     _cudart = _load_cudart()
     _cudart.cudaGetErrorName.restype = ctypes.c_char_p
     _cudart.cudaGetErrorString.restype = ctypes.c_char_p


Test patch: