NaN output in model after doing optimizer.step()

username: PATHIK_GHUGARE
cooked: <p>I am training a speech to text model, following is my training loop:</p>
<pre data-code-wrap="python"><code class="lang-python">for i in range(epochs):
    print(f"Epoch {i+1}/{epochs}")
    wer = 0
    mer = 0
    wil = 0 
    running_loss = 0
    for data in tqdm(train_dataloader):
        # Every data instance is an input + label pair
        inputs, labels, input_lengths, label_lengths = data
        inputs, labels = inputs.to(device, dtype=torch.float32), labels.to(device, dtype=torch.float32)

        print(torch.max(inputs), torch.min(inputs))

        # Zero your gradients for every batch!
        optimizer.zero_grad()

        # Make predictions for this batch
        inputs = inputs.permute(0,2,1) # (1, 1729, 400)
        outputs = model(inputs)
        print(outputs.shape) # 1, 1729, 29 [N, num_frames, probs]

        print(outputs)

        # Compute the loss and its gradients
        loss = criterion(outputs, labels, input_lengths, label_lengths)
        loss.backward()
        
        # Error metrics
        decoded_preds, decoded_targets = GreedyDecoder(outputs.transpose(0, 1), labels, label_lengths) # to debug model's output and ground truth
        measures = jiwer.compute_measures(decoded_preds, decoded_targets)
        wer += measures['wer']/inputs.shape[0]
        mer += measures['mer']/inputs.shape[0]
        wil += measures['wil']/inputs.shape[0]
        running_loss += loss.item()/inputs.shape[0]

        print(decoded_preds)
        print(decoded_targets)

        # Adjust learning weights
        optimizer.step()

    print("WER :", wer)
    print("MER :", mer)
    print("WIL :", wil)
    print("Loss :", running_loss)
    break
</code></pre>
<p>just to test I am using a <code>batch_size=1</code> in my dataloader and optimizer which I am using is <code>AdamW</code> with <code>lr=1e-4</code> and <code>criterion</code> as <code>CTCLoss()</code></p>
<p>but after first step, output of model becomes <code>nan</code> for some reason and I suspect that its happening because of the <code>optimizer.step()</code></p>
<p>Following is the output after first step:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5bb734c251556d5f93bd41dc95234a24a71cc81c.png" data-download-href="https://discuss.pytorch.org/uploads/default/5bb734c251556d5f93bd41dc95234a24a71cc81c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5bb734c251556d5f93bd41dc95234a24a71cc81c.png" alt="image" data-base62-sha1="d5lXXkK2stnyCSTryN3o178unjS" width="690" height="271" data-dominant-color="3D3D3D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1776×698 30 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why is this happening? and how can I resolve this issue?<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: clerkie
cooked: <p>What is the output of the print statements?</p>