Distributed training with GPU’s

username: JIAOJIAYUASD
cooked: <ul>
<li>/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [1,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.</li>
<li>/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [40,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.</li>
<li>/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [6,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.</li>
<li>Traceback (most recent call last):</li>
<li>File “main.py”, line 954, in </li>
<li>main()</li>
<li>File “main.py”, line 665, in main</li>
<li>optimizers=optimizers)</li>
<li>File “main.py”, line 773, in train_one_epoch</li>
<li>label_size=args.token_label_size)</li>
<li>File “/opt/conda/lib/python3.6/site-packages/tlt/data/mixup.py”, line 90, in mixup_target</li>
<li>y1 = get_labelmaps_with_coords(target, num_classes, on_value=on_value, off_value=off_value, device=device, label_size=label_size)</li>
<li>File “/opt/conda/lib/python3.6/site-packages/tlt/data/mixup.py”, line 64, in get_labelmaps_with_coords</li>
<li>num_classes=num_classes,device=device)</li>
<li>File “/opt/conda/lib/python3.6/site-packages/tlt/data/mixup.py”, line 12, in get_featuremaps</li>
<li>label_maps_topk_sizes[3]], 0, dtype=torch.float32 ,device=device)</li>
<li>RuntimeError: CUDA error: device-side assert triggered</li>
<li>terminate called after throwing an instance of ‘std::runtime_error’</li>
<li>what(): NCCL error in: /pytorch/torch/lib/c10d/…/c10d/NCCLUtils.hpp:136, unhandled cuda error, NCCL version 2.7.8</li>
</ul>

username: ptrblck
cooked: <p>The stacktrace points to an invalid indexing operation.<br>
You could try to rerun your script with <code>CUDA_LAUNCH_BLOCKING=1</code>, which should print the operation using the invalid index in the stacktrace directly. If that doesn’t work, try to use this env variable with a single GPU or run the script on the CPU to isolate the issue further.</p>

username: JIAOJIAYUASD
cooked: <p>Thanks for your reply. I did the operation like what you said, but I got the same debug report. Just like this :<br>
Train: 0 [   0/40036 (  0%)]  Loss: 10.430088 (10.4301)  Time: 4.107s,    7.79/s  (4.107s,    7.79/s)  LR: 1.000e-06  Data: 2.876 (2.876)<br>
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [25,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
Traceback (most recent call last):<br>
File “main.py”, line 949, in <br>
main()<br>
File “main.py”, line 664, in main<br>
optimizers=optimizers)<br>
File “main.py”, line 773, in train_one_epoch<br>
label_size=args.token_label_size)<br>
File “/opt/conda/lib/python3.6/site-packages/tlt/data/mixup.py”, line 90, in mixup_target<br>
y1 = get_labelmaps_with_coords(target, num_classes, on_value=on_value, off_value=off_value, device=device, label_size=label_size)<br>
File “/opt/conda/lib/python3.6/site-packages/tlt/data/mixup.py”, line 70, in get_labelmaps_with_coords<br>
device=device)<br>
File “/opt/conda/lib/python3.6/site-packages/tlt/data/mixup.py”, line 31, in get_label<br>
1).float().to(device),<br>
RuntimeError: CUDA error: device-side assert triggered</p>
<p>what’s more. I am running the code of volo #<a href="https://github.com/sail-sg/volo" rel="noopener nofollow ugc">https://github.com/sail-sg/volo</a><br>
with apex-amp, pytorch1.7.1, cuda10.1, torchvision0.8.2 in a single GPU.</p>

username: ptrblck
cooked: <p>Based on the stacktrace the blocking launch env variable doesn’t seem to be set or didn’t work properly, so you could either run the code on the CPU or synchronize manually the script (via <code>torch.cuda.synchronize()</code>) to further isolate the invalid index. Alternatively, using <code>assert</code> statements to check the input to indexing operations might also work.</p>