Device not recognized by nerfstudio

Shrikant_Arvavasu: <p>I am getting this error every time I am running the training command. I am using an HPC server with SLURM and cuda is loaded using <code>module load cuda cudnn</code> commands. I am running the script on a compute GPU node, but the GPU usage is 0%.</p>
<p>Traceback (most recent call last):<br>
File “/home/ashri/.conda/envs/nerfstudio/lib/python3.8/site-packages/multiprocess/process.py”, line 315, in _bootstrap<br>
self.run()<br>
File “/home/ashri/nerfstudio/nerfstudio/data/datamanagers/parallel_datamanager.py”, line 106, in run<br>
ray_bundle = ray_bundle.pin_memory()<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/tensor_dataclass.py”, line 273, in pin_memory<br>
return self._apply_fn_to_fields(lambda x: x.pin_memory())<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/tensor_dataclass.py”, line 303, in _apply_fn_to_fields<br>
new_fields = self._apply_fn_to_dict(<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/tensor_dataclass.py”, line 344, in _apply_fn_to_dict<br>
new_dict[f] = fn(v)<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/tensor_dataclass.py”, line 273, in <br>
return self._apply_fn_to_fields(lambda x: x.pin_memory())<br>
RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.</p>
<p>I have tried to install pytorch from source by setting TORCH_USE_CUDA_DSA=1, but it doesn’t help out. Any suggestion to solve this is appreciated.</p>

ptrblck: <p>Could you check if your current setup is able to detect the GPU in any other CUDA application (e.g. a test from the <code>CUDASamples</code>)?</p>

Shrikant_Arvavasu: <p>If I run torch.cuda.is_available() or torch.randn(shape, device = ‘cuda’) from ipython in the conda environment from the server, it doesn’t give any problem.</p>

ptrblck: <p>This would point to an environment issue and you should check which Python environment is used by default in your terminal and which one is used in Jupyter.</p>

Shrikant_Arvavasu: <p>I am using <code>conda activate nerfstudio</code> before running the code in an interactive session.</p>
<p>Traceback (most recent call last):<br>
File “/home/ashri/.conda/envs/nerfstudio/lib/python3.8/site-packages/multiprocess/process.py”, line 315, in _bootstrap<br>
self.run()<br>
File “/home/ashri/nerfstudio/nerfstudio/data/datamanagers/parallel_datamanager.py”, line 106, in run<br>
ray_bundle = ray_bundle.pin_memory()<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/tensor_dataclass.py”, line 273, in pin_memory<br>
return self._apply_fn_to_fields(lambda x: x.pin_memory())<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/tensor_dataclass.py”, line 303, in _apply_fn_to_fields<br>
new_fields = self._apply_fn_to_dict(<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/tensor_dataclass.py”, line 344, in _apply_fn_to_dict<br>
new_dict[f] = fn(v)<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/tensor_dataclass.py”, line 273, in <br>
return self._apply_fn_to_fields(lambda x: x.pin_memory())<br>
RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>╭─────────────── viser ───────────────╮<br>
│             ╷                       │<br>
│   HTTP      │ <a href="http://0.0.0.0:7007" rel="noopener nofollow ugc">http://0.0.0.0:7007</a>   │<br>
│   Websocket │ ws://0.0.0.0:7007     │<br>
│             ╵                       │<br>
╰─────────────────────────────────────╯<br>
[NOTE] Not running eval iterations since only viewer is enabled.<br>
Use --vis {wandb, tensorboard, viewer+wandb, viewer+tensorboard} to run with eval.<br>
No Nerfstudio checkpoint to load, so training from scratch.<br>
Disabled comet/tensorboard/wandb event writers<br>
^CTraceback (most recent call last):<br>
File “/home/ashri/nerfstudio/nerfstudio/scripts/train.py”, line 189, in launch<br>
main_func(local_rank=0, world_size=world_size, config=config)<br>
File “/home/ashri/nerfstudio/nerfstudio/scripts/train.py”, line 100, in train_loop<br>
trainer.train()<br>
File “/home/ashri/nerfstudio/nerfstudio/engine/trainer.py”, line 261, in train<br>
loss, loss_dict, metrics_dict = self.train_iteration(step)<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/profiler.py”, line 112, in inner<br>
out = func(*args, **kwargs)<br>
File “/home/ashri/nerfstudio/nerfstudio/engine/trainer.py”, line 496, in train_iteration<br>
_, loss_dict, metrics_dict = self.pipeline.get_train_loss_dict(step=step)<br>
File “/home/ashri/nerfstudio/nerfstudio/utils/profiler.py”, line 112, in inner<br>
out = func(*args, **kwargs)<br>
File “/home/ashri/nerfstudio/nerfstudio/pipelines/base_pipeline.py”, line 300, in get_train_loss_dict<br>
ray_bundle, batch = self.datamanager.next_train(step)<br>
File “/home/ashri/nerfstudio/nerfstudio/data/datamanagers/parallel_datamanager.py”, line 290, in next_train<br>
bundle, batch = self.data_queue.get()<br>
File “/home/ashri/.conda/envs/nerfstudio/lib/python3.8/site-packages/multiprocess/queues.py”, line 100, in get<br>
res = self._recv_bytes()<br>
File “/home/ashri/.conda/envs/nerfstudio/lib/python3.8/site-packages/multiprocess/connection.py”, line 219, in<br>
recv_bytes<br>
buf = self._recv_bytes(maxlength)<br>
File “/home/ashri/.conda/envs/nerfstudio/lib/python3.8/site-packages/multiprocess/connection.py”, line 417, in<br>
_recv_bytes<br>
buf = self._recv(4)<br>
File “/home/ashri/.conda/envs/nerfstudio/lib/python3.8/site-packages/multiprocess/connection.py”, line 382, in _recv<br>
chunk = read(handle, remaining)<br>
KeyboardInterrupt</p>
<p>Printing profiling stats, from longest to shortest duration in seconds<br>
Trainer.train_iteration: 55.9284<br>
VanillaPipeline.get_train_loss_dict: 55.9271</p>
<h1><a name="ipython-1" class="anchor" href="#ipython-1"></a>IPYTHON</h1>
<p>(nerfstudio) [ashri@gl1503 nerfstudio]$ ipython<br>
Python 3.8.19 (default, Mar 20 2024, 19:58:24)<br>
Type ‘copyright’, ‘credits’ or ‘license’ for more information<br>
IPython 8.12.3 – An enhanced Interactive Python. Type ‘?’ for help.</p>
<p>In [1]: import torch</p>
<p>In [2]: tensor = torch.randn((4,3), device = ‘cuda’)</p>
<p>In [3]: print(tensor)<br>
tensor([[-0.0186,  0.0662,  0.0233],<br>
[ 0.8250, -1.3138,  0.0955],<br>
[ 2.4501, -1.4790, -0.2986],<br>
[-0.2575,  1.6024, -1.0451]], device=‘cuda:0’)</p>