RuntimeError("No GPUs available.")

r00bi: <p>Hi,<br>
I was running my code on 8 gpus, and changing my code related to reading data. Suddenly after few runs I’ve stared getting this error:</p>
<p>File “~anaconda3/envs/_new/lib/python3.10/site-packages/pytorch_lightning/utilities/argparse.py”, line 348, in insert_env_defaults<br>
return fn(self, **kwargs)<br>
File “~/anaconda3/envs/_new/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 419, in <strong>init</strong><br>
self._accelerator_connector = AcceleratorConnector(<br>
File “~/anaconda3/envs/_new/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py”, line 205, in <strong>init</strong><br>
self._set_parallel_devices_and_init_accelerator()<br>
File “~/anaconda3/envs/_new/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py”, line 566, in _set_parallel_devices_and_init_accelerator<br>
self._set_devices_flag_if_auto_select_gpus_passed()<br>
File “~/anaconda3/envs/_new/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py”, line 583, in _set_devices_flag_if_auto_select_gpus_passed<br>
self._devices_flag = pick_multiple_gpus(<br>
File “~/anaconda3/envs/_new/lib/python3.10/site-packages/pytorch_lightning/tuner/auto_gpu_select.py”, line 54, in pick_multiple_gpus<br>
picked.append(pick_single_gpu(exclude_gpus=picked, _show_deprecation=False))<br>
File “~/anaconda3/envs/_new/lib/python3.10/site-packages/pytorch_lightning/tuner/auto_gpu_select.py”, line 96, in pick_single_gpu<br>
raise RuntimeError(“No GPUs available.”)</p>
<p>though the gpus are available and work for other code.<br>
Could you please let me how I can fix it?<br>
Thanks!</p>

fduwjj: <p>What is the output of “torch.cuda.is_available()” in your case? Have you started training yet?</p>

r00bi: <aside class="quote no-group" data-username="fduwjj" data-post="2" data-topic="180080">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fduwjj/48/47855_2.png" class="avatar"> fduwjj:</div>
<blockquote>
<p>hat is the output of “torch.cuda.is_available()” in your case? Have you started training yet?</p>
</blockquote>
</aside>
<p>Previously it was True, but for some reason it returned False, but I learned that the issue is the  CUDA_VISIBLE_DEVICES, after I set it to 1, it could recognize one of the gpus:</p>
<pre><code class="lang-auto">export CUDA_VISIBLE_DEVICES=1
</code></pre>
<p>However, when I set to all gups: ```<br>
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7</p>
<pre><code class="lang-auto">getting the same error:
lib/python3.10/site-packages/pytorch_lightning/tuner/auto_gpu_select.py", line 96, in pick_single_gpu
    raise RuntimeError("No GPUs available.")
RuntimeError: No GPUs available.</code></pre>

fduwjj: <p>How did you start your training job on 8GPUs?</p>

r00bi: <p>There are 8 16G GPUS:<br>
import pytorch_lightning as pl</p>
<p>trainer = pl.Trainer(<br>
accelerator=“auto”,<br>
auto_select_gpus=True,<br>
callbacks=callbacks,<br>
devices=_get_devices(),<br>
max_epochs=config[“max_epochs”],<br>
strategy=_get_strategy(),<br>
)</p>
<p>trainer.fit(<br>
model, train_loader.train_dataloader(), val_loader.val_dataloader()<br>
)</p>