GPU startup is way too slow!

sajastu: <p>Hi,</p>
<p>I’m using NVIDIA RTX A6000 GPUs to run an NLP task (using Transformers library). The problem is that when I want to perform training in a multi-GPU environment (i.e., dual GPU), the system just gets stuck before training starts and makes no progress as far as it shows in the log. I’m not sure which part is problematic since I recently have upgraded my GPU cluster. – I even managed to see when it progresses the training, but after 5-6 hours, as I saw no change, I stopped the script.</p>
<p>Following are the system and packages info:</p>
<blockquote>
<p>NVIDIA-SMI 510.60.02<br>
Driver Version: 510.60.02<br>
CUDA Version: 11.6<br>
Pytorch: 1.8.2<br>
Arch version: SM_75, compute_75</p>
</blockquote>

ptrblck: <p>Based on your output I assume you’ve built PyTorch from source for Turing GPUs (architectures <code>sm_75</code>).<br>
If so, add <code>sm_80</code> and <code>sm_86</code> to your build and it should work on your Ampere GPU.</p>

sajastu: <p>Thanks for your answer. I’m actually not installing pytorch from source. Do I need to install it from source to enable this? If so, how can I add sm_80 and sm_86 to my build? I couldn’t find any documentation on this.</p>

ptrblck: <p>No, you don’t need to build from source to enable your Ampere GPU and I’m not sure how to interpret:</p>
<aside class="quote no-group" data-username="sajastu" data-post="1" data-topic="147956">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sajastu/48/14519_2.png" class="avatar"> sajastu:</div>
<blockquote>
<p>Arch version: SM_75, compute_75</p>
</blockquote>
</aside>
<p>in this case.<br>
The <code>1.8.2+cu111</code> binaries are built for your compute capability and work fine in my setup (using the same GPU architecture):</p>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.8.2+cu111'
&gt;&gt;&gt; torch.cuda.get_arch_list()
['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
&gt;&gt;&gt; torch.randn(1).cuda()
tensor([1.5997], device='cuda:0')
</code></pre>

sajastu: <p>Thanks, your provided code works fine on my machine as well. Just checked it and saw it includes sm_80 and sm_86 as well. The problem, though, still exists in multi-GPU environment. Here is the running logs of my code with the Transformers library.</p>
<p>code:<br>
<code>python3 -m torch.distributed.launch --nproc_per_node=2 training_script.py --[args]</code></p>
<p>logs:</p>
<blockquote>
<p>[INFO|modeling_utils.py:1431] 2022-04-03 10:15:19,610 &gt;&gt; loading weights file <a href="https://huggingface.co/allenai/led-base-16384/resolve/main/pytorch_model.bin" rel="noopener nofollow ugc">https://huggingface.co/allenai/led-base-16384/resolve/main/pytorch_model.bin</a> from cache at /home/sajad/.cache/huggingface/transformers/c8f7e4603efbc329ce921b34057d78880dead50f45b2a1648b3a06ca6eb17f51.201222b06d46289037a8dccc57548abc8eb81ba042d3762214ac15c9691ff8c7<br>
[INFO|modeling_utils.py:1698] 2022-04-03 10:15:21,212 &gt;&gt; All model checkpoint weights were used when initializing LEDForConditionalGeneration.</p>
<p>[INFO|modeling_utils.py:1706] 2022-04-03 10:15:21,213 &gt;&gt; All the weights of LEDForConditionalGeneration were initialized from the model checkpoint at allenai/led-base-16384.<br>
If your task is similar to the task the model of the checkpoint was trained on, you can already use LEDForConditionalGeneration for predictions without further training.<br>
04/03/2022 10:15:21 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sajad/.cache/huggingface/datasets/json/default-528e7879b13129b0/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b/cache-4067e49ab36511c2.arrow<br>
chianti:24372:24372 [0] NCCL INFO Bootstrap : Using enp8s0:192.168.10.202&lt;0&gt;<br>
chianti:24372:24372 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>chianti:24372:24372 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
chianti:24372:24372 [0] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.10.202&lt;0&gt;<br>
chianti:24372:24372 [0] NCCL INFO Using network Socket<br>
NCCL version 2.10.3+cuda11.1<br>
chianti:24373:24373 [1] NCCL INFO Bootstrap : Using enp8s0:192.168.10.202&lt;0&gt;<br>
chianti:24373:24373 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<h1>chianti:24373:24373 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
chianti:24373:24373 [1] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.10.202&lt;0&gt;<br>
chianti:24373:24373 [1] NCCL INFO Using network Socket<br>
chianti:24373:24420 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] -1/-1/-1-&gt;1-&gt;0<br>
chianti:24372:24419 [0] NCCL INFO Channel 00/02 :    0   1<br>
chianti:24372:24419 [0] NCCL INFO Channel 01/02 :    0   1<br>
chianti:24372:24419 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1<br>
chianti:24372:24419 [0] NCCL INFO Setting affinity for GPU 0 to ffffff<br>
chianti:24372:24419 [0] NCCL INFO Channel 00 : 0[a000] → 1[42000] via P2P/IPC<br>
chianti:24373:24420 [1] NCCL INFO Channel 00 : 1[42000] → 0[a000] via P2P/IPC<br>
chianti:24372:24419 [0] NCCL INFO Channel 01 : 0[a000] → 1[42000] via P2P/IPC<br>
chianti:24373:24420 [1] NCCL INFO Channel 01 : 1[42000] → 0[a000] via P2P/IPC<br>
chianti:24373:24420 [1] NCCL INFO Connected all rings<br>
chianti:24373:24420 [1] NCCL INFO Connected all trees<br>
chianti:24373:24420 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512<br>
chianti:24373:24420 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
chianti:24372:24419 [0] NCCL INFO Connected all rings<br>
chianti:24372:24419 [0] NCCL INFO Connected all trees<br>
chianti:24372:24419 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512<br>
chianti:24372:24419 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
chianti:24373:24420 [1] NCCL INFO comm 0x7ff580002f70 rank 1 nranks 2 cudaDev 1 busId 42000 - Init COMPLETE<br>
chianti:24372:24419 [0] NCCL INFO comm 0x7f4eb4002f70 rank 0 nranks 2 cudaDev 0 busId a000 - Init COMPLETE<br>
chianti:24372:24372 [0] NCCL INFO Launch mode Parallel<br>
[E ProcessGroupNCCL.cpp:719] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804805 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:719] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804805 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:406] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804805 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:406] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804805 milliseconds before timing out.<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 24372) of binary: /home/sajad/anaconda3/envs/myenv/bin/python3<br>
Traceback (most recent call last):<br>
File “/home/sajad/anaconda3/envs/myenv/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/home/sajad/anaconda3/envs/myenv/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/sajad/anaconda3/envs/myenv/lib/python3.8/site-packages/torch/distributed/launch.py”, line 193, in <br>
main()<br>
File “/home/sajad/anaconda3/envs/myenv/lib/python3.8/site-packages/torch/distributed/launch.py”, line 189, in main<br>
launch(args)<br>
File “/home/sajad/anaconda3/envs/myenv/lib/python3.8/site-packages/torch/distributed/launch.py”, line 174, in launch<br>
run(args)<br>
File “/home/sajad/anaconda3/envs/myenv/lib/python3.8/site-packages/torch/distributed/run.py”, line 715, in run<br>
elastic_launch(<br>
File “/home/sajad/anaconda3/envs/myenv/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/sajad/anaconda3/envs/myenv/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 245, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2>examples/pytorch/summarization/run_summarization.py FAILED</h2>
<h2>Failures:<br>
[1]:<br>
time      : 2022-04-03_10:45:32<br>
host      : chianti<br>
rank      : 1 (local_rank: 1)<br>
exitcode  : -6 (pid: 24373)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 6 (SIGABRT) received by PID 24373</h2>
<h1>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2022-04-03_10:45:32<br>
host      : chianti<br>
rank      : 0 (local_rank: 0)<br>
exitcode  : -6 (pid: 24372)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 6 (SIGABRT) received by PID 24372</h1>
</blockquote>

sajastu: <p>Searching over the internet, I feel like my problem kind of relates to this issue: <a href="https://github.com/pytorch/pytorch/issues/73790#issuecomment-1059662412" class="inline-onebox" rel="noopener nofollow ugc">A6000 Docker Ligthning ddp NCCL WARN Failed to open libibverbs.so · Issue #73790 · pytorch/pytorch · GitHub</a>. It does seem relevant as I’m also having this problem in a multi-GPU env with A6000 GPUs. As stands, the hack is to disable IOMMU as it locks up the system when used with multi-GPUs. I will try this solution and report the result here. Meanwhile, do you have any other ideas of why this is happening?</p>

ptrblck: <p>Your original description doesn’t really fit the linked issue, as you’ve mentioned your “startup is way too slow” (so I assume your script works after a while but just needs a “warmup” time for unknown reasons) while the linked issue mentions a hang, which never starts to execute the code.<br>
If you are hitting indeed the second issue, then yes: disable IOOMU and rerun the code.<br>
Also, is your code working fine in a single GPU use case?</p>

sajastu: <p>Yes, as I mentioned in the question, the training code gets stuck and never starts the training; it even does not exit the program with any error codes and so forth, which I assume is what the “hanging” thing is. My code works perfectly fine in single GPU use case. The only problem that I have is with multi-GPU cases. Just to confirm what I have understood, I should disable IOOMU on BIOS, right?</p>

sajastu: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your kind suggestion. I could finally get it fixed by disabling IOMMU in BIOS.</p>

ptrblck: <p>Great, thanks for the update!</p>

ziming_zhu: <p>Hello, I have the same problem, but my running platform is on a cluster, not a personal computer, so I can’t disable IOMMU. I would like to ask if there are other ways. The graphics card is eight A100</p>