RuntimeError: CUDA unknown error - Setting available devices to be zero

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I’m currently submitting a few scripts to a remote server with a few different GPUs and I’ve noticed that in some cases some jobs will fail due to this <code>CUDA unknown error</code>. The GPUs involved are specificied to use CUDA 11.0 and the PyTorch installation is 1.7.1+CUDA11. This error has emerged in few different ways but only when I called CUDA in some way. So, for example, moving my model from CPU to GPU will result in the same error (sometimes). The driver version is 450.66.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "~/run.py", line 28, in &lt;module&gt;
    device_name = torch.cuda.get_device_name(torch.cuda.current_device())
  File "~/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 366, in current_device
    _lazy_init()
  File "~/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 172, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.
</code></pre>
<p>Is there any particular way to diagnose the error so that I can resolve it? Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>This issue sounds more like a setup issue than a PyTorch error, so I would recommend to check the setup with the server admin and also take a look at <code>dmesg</code> (in particular search for any <code>xid</code> entries, which could provide more infromation, why CUDA is failing).</p>

username: AlphaBetaGamma96
cooked: <p>Do you have any references for further reading with this? I’ve never used <code>dmesg</code> and <code>xid</code> before to debug a CUDA install! Thank you once again! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p><a href="https://docs.nvidia.com/deploy/xid-errors/index.html">This document</a> gives you more information about these error codes.</p>

username: AlphaBetaGamma96
cooked: <p>Hello again! I’ve been briefly reading through the document you sent me and I’m not 100% sure how to proceed with solving this error. It states in the document you shared, that the Xid entries are located at <code>/var/log/messages/</code> however that directory does not exist. Is there some other command that needs to be run beforehand?</p>
<p>Also, I ran this command <code>dmesg | grep -e 'NVRM: Xid'</code> to see if any <code>Xid</code> entries appear from <code>dmesg</code> and it returns nothing. The only error that appears is a series of <code>nfs: RPC call returned error 13</code> errors, but that is all. Could this be a potential issue?</p>
<p>Also, also, could the type of card be an issue as well? Some cards are GTX 745 cards whereas some are more modern Quadro cards.</p>
<p>Thank you for all the help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="5" data-topic="122250">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>Also, also, could the type of card be an issue as well? Some cards are GTX 745 cards whereas some are more modern Quadro cards.</p>
</blockquote>
</aside>
<p>Yes, this could play a role in the issue, but it’s hard to tell without a proper error message (<code>unknown error</code> is unfortunately not very helpful <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> ) and apparently you are unable to see any <code>xid</code>s.</p>

username: OoOqn
cooked: <p>Hi. You could try to reboot the remote PC. The problem may raise because GPU drivers updated recently without rebooting <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: AlphaBetaGamma96
cooked: <p>It does seem like a bit of a problem! Is there anything else that comes to mind or am I out of luck? <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>Also, I was wondering if I could ask another question with some errors I get? For some reason I seem to get an issue with loading my model (occasionally).</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "~/main.py", line 145, in &lt;module&gt;
    state_dict = torch.load(f=model_path_pt, map_location=torch.device(device))
  File "~/.local/lib/python3.6/site-packages/torch/serialization.py", line 595, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "~/.local/lib/python3.6/site-packages/torch/serialization.py", line 764, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

Traceback (most recent call last):
  File "~/main.py", line 145, in &lt;module&gt;
    state_dict = torch.load(f=model_path_pt, map_location=torch.device(device))
  File "~/.local/lib/python3.6/site-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "~/.local/lib/python3.6/site-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
  File "~/.local/lib/python3.6/site-packages/torch/serialization.py", line 845, in persistent_load
    load_tensor(data_type, size, key, _maybe_decode_ascii(location))
  File "~/.local/lib/python3.6/site-packages/torch/serialization.py", line 833, in load_tensor
    storage = zip_file.get_storage_from_record(name, size, dtype).storage()
RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading file data/67511648: file read failed
</code></pre>
<p>For the first model it seems that the file is just 0 mb in size, is that correct? I only say this from reading this thread on stackoverflow <a href="https://stackoverflow.com/questions/61215873/eoferror-ran-out-of-input-using-torch-load" rel="noopener nofollow ugc">here</a>. For the second one, I’m not 100% sure what’s wrong. I did read you’re previous answer <a href="https://discuss.pytorch.org/t/error-on-torch-load-pytorchstreamreader-failed/95103">here</a> but I’m saving everything within a dictionary rather than saving the model directly like this…</p>
<pre><code class="lang-auto">torch.save({'epoch':preepoch,
            'model_state_dict':net.state_dict(),
            'optim_state_dict':optim.state_dict(),
            'loss':mean_preloss,
            'chains':sampler.chains}, model_path_pt)
</code></pre>
<p>and then loaded with</p>
<pre><code class="lang-auto">state_dict = torch.load(f=model_path_pt, map_location=torch.device(device))
start=state_dict['epoch']+1
net.load_state_dict(state_dict['model_state_dict'])
optim.load_state_dict(state_dict['optim_state_dict'])
loss = state_dict['loss']
sampler.chains = state_dict['chains']
</code></pre>
<p>Thank you!</p>
<p>Edit: A follow up question to the <code>PytorchStreamReader</code> error, I save my model each epoch and each epoch takes around 0.3s to do. Is it advisable to save at each epoch or to save every n-th epoch?. Could this be causing the issue with reading a file each 0.3s? Because the error does vary a bit sometimes it’s <code>failed finding central directory</code>, <code>invalid header or archive is corrupted</code>, or <code>file read failed</code>!</p>

username: AlphaBetaGamma96
cooked: <p>Thank you, <a class="mention" href="/u/oooqn">@OoOqn</a>! I tried that but not much changed! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>You could create a topic in the NVIDIA board following <a href="https://forums.developer.nvidia.com/t/if-you-have-a-problem-please-read-this-first/27131">these steps</a> to provide a full log, which might help to isolate the issue further.</p>