akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Segfault, double freee, corrupted linked-list, etc. in loss.backward()

stellarpower: <p>Hi,</p>
<p>I’ve installed pytorch into a mamba environment through the official installer.</p>
<p>I have checked my prediction and loss tensors for infinity/NaNs, and the forwards loss is a real number. I’ve disabled CUDA for now. I get a variety of memory errors during the backwards pass (<code>single_loss.backwards()</code>) - this is the training loop:</p>
<pre><code class="lang-auto">for i in range(epochs):
    epoch_loss = 0
    for inputs, groundTruth in dataloader:
        optimizer.zero_grad()
        y_pred = model(inputs)
        testForBadValues(y_pred,"Training forward pass outputs")

        single_loss = loss_function(y_pred, groundTruth)
        
        # The backwards pass does not handle nans properly, it seems.
        testForBadValues(single_loss, "Training loss")

        single_loss.backward()
        epoch_loss += single_loss
        optimizer.step()
    train_losses.append(epoch_loss)

    model.eval()
    val_loss = 0
    for inputs, groundTruth in val_dataloader:
        y_pred = model(inputs)
        single_loss = loss_function(y_pred, groundTruth)
        val_loss += single_loss
    val_losses.append(val_loss)
    model.train()

    # Save model if validation loss has decreased
    if val_loss &lt; best_val_loss:
        best_val_loss = val_loss
        torch.save(model.state_dict(), 'best_model.pth')
</code></pre>
<p>Before I paste a lot of code, is there anything I can do to debug this? I see mention of similar problems come up a number of times through Google, but each seems to be a different corner case, and especially suspicious if I’ve hidden the GPU through <code>CUDA_VISIBLE_DEVICES</code>. I’ve mostly been using Keras (am modifying someone else’s code), so am not massively familiar with Torch. It seems the official installation does not include debugging symbols, so I can’t inspect my backtrace in much detail. I’d rather not spend hours building from source if I can avoid it. Is there any other way to get more information about the cause of the crash? Exit code is always 245</p>
<p>Thanks</p>
ValueError: Default process group has not been initialized, please make sure to call init_process_group.

CMS: <p>Traceback (most recent call last):<br>
File “/content/xl-sum/seq2seq/pipeline.py”, line 517, in<br>
main()<br>
File “/content/xl-sum/seq2seq/pipeline.py”, line 293, in main</p>
<ul>
<li>(torch.distributed.get_world_size() if training_args.local_rank != -1 else 1)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py”, line 1555, in get_world_size<br>
return _get_group_size(group)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py”, line 836, in _get_group_size<br>
default_pg = _get_default_group()<br>
File “/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py”, line 977, in _get_default_group<br>
raise ValueError(<br>
ValueError: Default process group has not been initialized, please make sure to call init_process_group.</li>
</ul>

ptrblck: <p>Did you initialize the default ProcessGroup in your <code>DDP</code> use case as described <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">here</a>?</p>