Test loss and dice coefficient giving nan result

Anastasia_Berlianna: <p>i build unet model to segment diabetic retinopathy lesion, but when i did the training with 30 epochs, it give nan value in test loss and dice coefficient</p>
<pre><code class="lang-auto">startTime = time.time()
batch_size= 2

train_dataset = ProcessDataset(train_x, train_y)
test_dataset = ProcessDataset(test_x, test_y)

train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, 
num_workers=os.cpu_count(), pin_memory=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False, 
num_workers=os.cpu_count(), pin_memory=True)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = Build_Unet()
model.to(device)

trainSteps = len(train_dataset) // batch_size
testSteps = len(test_dataset) // batch_size

H = {"train_loss": [], "test_loss": [], "accuracy": [], "dice": []}

criterion = nn.BCEWithLogitsLoss()
criterion.to(device)

optimizer = optim.Adam(model.parameters(), lr=0.001)

num_epochs = 30

for epoch in range(num_epochs):

    model.train()

    totalTrainLoss = 0
    totalTestLoss = 0

    for i, (data, target) in enumerate(train_loader):

        data, target = data.to(device), target.to(device)
        target = target.unsqueeze(1) 

        output = model(data)
        loss = criterion(output, target)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        totalTrainLoss += loss

    with torch.no_grad():

        model.eval()

        total_dice = 0

        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            target = target.unsqueeze(1) 

            output = model(data)

            loss = criterion(output, target)
            totalTestLoss += loss

            pred = torch.round(output)

            dice = compute_meandice(pred, target, include_background=False)
            total_dice += dice

    avg_dice = total_dice / len(test_loader)
    avgTrainLoss = totalTrainLoss / trainSteps
    avgTestLoss = totalTestLoss / testSteps

    H["train_loss"].append(avgTrainLoss.cpu().detach().numpy())
    H["test_loss"].append(avgTestLoss.cpu().detach().numpy())
    H["dice"].append(avg_dice)

    print("EPOCH: {}/{}".format(epoch + 1, num_epochs))
    print("Train loss: {:.4f} | Test loss: {:.4f}".format(
           avgTrainLoss, avgTestLoss))
    print("Dice coefficient: {:.4f}".format(avg_dice.mean().item()))

endTime = time.time()
print("[INFO] total time taken to train the model: {:.2f}s".format(
    endTime - startTime))
</code></pre>
<p>here i want to print training loss, test loss, and dice coefficient to see my model performance, but i got stuck in test loss and dice coeficient giving nan value like this</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/650290797aa604087ae1a106a2d1e4c651659fb2.png" alt="image" data-base62-sha1="epzIymHVpnzLDs2ypj3mo1VwpVg" width="246" height="97"></p>
<p>since i’m newbie in pytorch, anyone know how to fix it? or is there something strange in my code? thankyou in adanvance!</p>

KFrank: <p>Hi Anastasia!</p>
<aside class="quote no-group" data-username="Anastasia_Berlianna" data-post="1" data-topic="175372" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anastasia_berlianna/48/58506_2.png" class="avatar"> Anastasia_Berlianna:</div>
<blockquote>
<p>it give nan value in test loss and dice coefficient</p>
</blockquote>
</aside>
<p>First some context:  <code>nan</code> is a “special” floating-point number.  It means<br>
“not a number.”  It appears as the result of certain ill-defined mathematical<br>
operations such as zero divided by zero or infinity minus infinity.</p>
<p>It also has the property that any operation on a <code>nan</code> will result in another<br>
<code>nan</code>.</p>
<p>So, if an element of one of the weights in your model is <code>nan</code>, then the<br>
output of your model will (in most cases) be <code>nan</code>.</p>
<p>Now some suggestions on how to track down the source of your <code>nan</code>:</p>
<p>That your <code>train loss</code> is not <code>nan</code> suggests that your model doesn’t have<br>
any <code>nan</code>s in it.</p>
<p>So first check that your test data (that you input to your model) doesn’t<br>
have any <code>nan</code>s in it.  If not, then start passing samples from <code>test_dataset</code><br>
through your model and check whether the <code>output</code> has any <code>nan</code>s.  Note<br>
that any single <code>nan</code> in the <code>output</code> from any sample from <code>test_dataset</code><br>
will result in <code>test_loss</code> becoming <code>nan</code>.</p>
<p>Beware that if <code>len (test_dataset)</code> is less than <code>batch_size</code> then<br>
<code>testSteps = len(test_dataset) // batch_size</code> will round down to<br>
zero, and <code>avgTestLoss = totalTestLoss / testSteps</code> will be <code>0.0 / 0</code><br>
which will be <code>nan</code>.</p>
<p>Try these steps to see if you can locate where the <code>nan</code>s are creeping in<br>
and feel free to post any follow-up questions.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Anastasia_Berlianna: <p>hi, thankyou for your time. you suggest to check my test data, how can i check that? my test dataset is contains of 27 images, so i assume that testSteps won’t be round down to zero. could you provide me some solution? thanks!</p>

KFrank: <p>Hi Anastasia!</p>
<aside class="quote no-group" data-username="Anastasia_Berlianna" data-post="3" data-topic="175372" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anastasia_berlianna/48/58506_2.png" class="avatar"> Anastasia_Berlianna:</div>
<blockquote>
<p>how can i check that?</p>
</blockquote>
</aside>
<p>You could do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">            if  data.isnan().any():
                 print ('found a nan in data.')
            output = model(data)
            if  output.isnan().any():
                 print ('found a nan in output.')
</code></pre>
<p>and do the same for <code>target</code> and <code>loss</code>, if necessary.</p>
<p>Best.</p>
<p>K. Frank</p>

Anastasia_Berlianna: <p>hi, thanks for your advice! now my test loss already had a value, but my dice coefficient still nan. i try to do your advice but it didn’t print anything so i assume that my data and target are definitely okay.</p>
<p>97%|█████████▋| 29/30 [02:02&lt;00:04, 4.31s/it]</p>
<p>EPOCH: 29/30 Train loss: 0.0289 | Test loss: 0.0508 Dice coefficient: nan</p>
<p>100%|██████████| 30/30 [02:06&lt;00:00, 4.21s/it]</p>
<p>EPOCH: 30/30 Train loss: 0.0299 | Test loss: 0.0442 Dice coefficient: nan</p>

KFrank: <p>Hi Anastasia!</p>
<aside class="quote no-group" data-username="Anastasia_Berlianna" data-post="5" data-topic="175372" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anastasia_berlianna/48/58506_2.png" class="avatar"> Anastasia_Berlianna:</div>
<blockquote>
<p>my dice coefficient still nan.</p>
</blockquote>
</aside>
<p>The Dice coefficient is a normalized count of the number of true positives<br>
your model predicts.  If neither your ground truth (your <code>target</code>) nor your<br>
(rounded) predictions (your <code>pred</code>) contain any positives for the sample<br>
(or batch of samples) in question, then the formula for the Dice coefficient<br>
will give <code>0 / 0</code> which will become <code>nan</code>.</p>
<p>If your function <code>compute_meandice()</code> doesn’t protect against such a<br>
possibility, then it could return <code>nan</code> and pollute your running Dice score,<br>
<code>total_dice</code>.</p>
<p>If this doesn’t happen very often, you could just use <code>0.0</code> for the value<br>
of <code>dice</code> when <code>nan</code> occurs.  If it happens with some frequency, you might<br>
want to leave such samples out of your <code>avg_dice</code> computation:</p>
<pre data-code-wrap="python"><code class="lang-python">avg_dice = total_non_nan_dice / count_non_nan_dice
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Anastasia_Berlianna: <p>ah thanks! after i rechecked, it is because centercropping that crop the mask, so the mask is just black background</p>