RuntimeWarning: invalid value encountered in double_scalars

username: fyy
cooked: <p>when I train my network ,my loss is Falling<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82f622faeed8aa51c6090b606a21bc8f3129fdd5.png" data-download-href="https://discuss.pytorch.org/uploads/default/82f622faeed8aa51c6090b606a21bc8f3129fdd5" title="4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82f622faeed8aa51c6090b606a21bc8f3129fdd5_2_690x350.png" alt="4" data-base62-sha1="iGxpOWYEXh9SmS4kHKHipDfgSK9" width="690" height="350" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82f622faeed8aa51c6090b606a21bc8f3129fdd5_2_690x350.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82f622faeed8aa51c6090b606a21bc8f3129fdd5_2_1035x525.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82f622faeed8aa51c6090b606a21bc8f3129fdd5_2_1380x700.png 2x" data-dominant-color="F9FBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">1577×801 42.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
but my acc is not  rising and get a RuntimeWarning<br>
<code>Epoch [59/1000], loss: 0.6919251084, training_acc: 0.499944</code><br>
<code>Epoch [59/1000], loss: 0.6918576956, training_acc: 0.499945</code><br>
<code>Epoch [59/1000], loss: 0.6918291450, training_acc: 0.499945</code><br>
<code>Epoch [59/1000], loss: 0.6918825507, training_acc: 0.499948</code><br>
<code>Epoch [59/1000], loss: 0.6917557716, training_acc: 0.499946</code><br>
<code>Epoch [60/1000], loss: 0.6918535829, training_acc: 0.499949</code><br>
<code>Epoch [60/1000], loss: 0.6918678284, training_acc: 0.499949</code><br>
<code>Epoch [60/1000], loss: 0.6917811632, training_acc: 0.499950</code><br>
<code>Epoch [60/1000], loss: 0.6917777658, training_acc: 0.499949</code><br>
<code>Epoch [60/1000], loss: 0.6916995645, training_acc: 0.499948</code><br>
<code>Epoch [61/1000], loss: 0.6917661428, training_acc: 0.499947</code><br>
<code>Epoch [61/1000], loss: 0.6917319894, training_acc: 0.499946</code><br>
<code>Epoch [61/1000], loss: 0.6918124557, training_acc: 0.499948</code><br>
<code>Epoch [61/1000], loss: 0.6916714907, training_acc: 0.499948</code><br>
<code>Epoch [61/1000], loss: 0.6917373538, training_acc: 0.499948</code><br>
<code>Epoch [62/1000], loss: 0.6915817857, training_acc: 0.499944</code><br>
<code>Epoch [62/1000], loss: 0.6917345524, training_acc: 0.499946</code><br>
<code>Epoch [62/1000], loss: 0.6917485595, training_acc: 0.499948</code><br>
<code>Epoch [62/1000], loss: 0.6917622685, training_acc: 0.499952</code><br>
<code>Epoch [62/1000], loss: 0.6916363239, training_acc: 0.499950</code><br>
<code>Epoch [63/1000], loss: 0.6916769147, training_acc: 0.499954</code><br>
<code>Epoch [63/1000], loss: 0.6916103959, training_acc: 0.499953</code><br>
<code>Epoch [63/1000], loss: 0.6916091442, training_acc: 0.499952</code><br>
<code>Epoch [63/1000], loss: 0.6916871071, training_acc: 0.499958</code><br>
<code>Epoch [63/1000], loss: 0.6915804744, training_acc: 0.499957</code><br>
<code>Epoch [64/1000], loss: 0.6916241050, training_acc: 0.499961</code><br>
<code>Epoch [64/1000], loss: 0.6915611625, training_acc: 0.499966</code><br>
<code>Epoch [64/1000], loss: 0.6915152669, training_acc: 0.499966</code><br>
<code>Epoch [64/1000], loss: 0.6915746927, training_acc: 0.499966</code><br>
<code>Epoch [64/1000], loss: 0.6916203499, training_acc: 0.499966</code><br>
<code>Epoch [65/1000], loss: 0.6915393472, training_acc: 0.499973</code><br>
<code>Epoch [65/1000], loss: 0.6916345358, training_acc: 0.499968</code><br>
<code>Epoch [65/1000], loss: 0.6914507747, training_acc: 0.499969</code><br>
<code>Epoch [65/1000], loss: 0.6915865541, training_acc: 0.499971</code><br>
<code>Epoch [65/1000], loss: 0.6914009452, training_acc: 0.499969</code><br>
<code>Epoch [66/1000], loss: 0.6915102601, training_acc: 0.499980</code><br>
<code>Epoch [66/1000], loss: 0.6915132999, training_acc: 0.499984</code><br>
<code>Epoch [66/1000], loss: 0.6914609671, training_acc: 0.499987</code><br>
<code>Epoch [66/1000], loss: 0.6913943887, training_acc: 0.499988</code><br>
<code>Epoch [66/1000], loss: 0.6914426088, training_acc: 0.499988</code><br>
<code>F:/experiment_code/U-net/train_2.py:146: RuntimeWarning: invalid value encountered in double_scalars   p = TP / (TP + FP)</code><br>
<code>Epoch [67/1000], loss: 0.6912648678, training_acc: 0.500000</code><br>
<code>Epoch [67/1000], loss: 0.6914439797, training_acc: 0.500000</code><br>
<code>Epoch [67/1000], loss: 0.6913767457, training_acc: 0.500000</code><br>
<code>Epoch [67/1000], loss: 0.6914988756, training_acc: 0.499998</code><br>
<code>Epoch [67/1000], loss: 0.6914470196, training_acc: 0.499999</code><br>
<code>Epoch [68/1000], loss: 0.6913981438, training_acc: 0.500000</code><br>
<code>Epoch [68/1000], loss: 0.6914529800, training_acc: 0.500000</code><br>
<code>Epoch [68/1000], loss: 0.6912329793, training_acc: 0.500000</code><br>
<code>Epoch [68/1000], loss: 0.6913060546, training_acc: 0.500000</code><br>
<code>Epoch [68/1000], loss: 0.6913271546, training_acc: 0.500000</code><br>
<code>Epoch [69/1000], loss: 0.6913089156, training_acc: 0.500000</code><br>
<code>Epoch [69/1000], loss: 0.6913014650, training_acc: 0.500000</code><br>
<code>Epoch [69/1000], loss: 0.6913487315, training_acc: 0.500000</code><br>
<code>Epoch [69/1000], loss: 0.6912917495, training_acc: 0.500000</code><br>
<code>Epoch [69/1000], loss: 0.6911794543, training_acc: 0.500000</code><br>
<code>Epoch [70/1000], loss: 0.6913054585, training_acc: 0.500000</code><br>
<code>Epoch [70/1000], loss: 0.6910668015, training_acc: 0.500000</code><br>
<code>Epoch [70/1000], loss: 0.6912170053, training_acc: 0.500000</code><br>
<code>Epoch [70/1000], loss: 0.6912853718, training_acc: 0.500000</code><br>
<code>Epoch [70/1000], loss: 0.6912366152, training_acc: 0.500000</code><br>
<code>Epoch [71/1000], loss: 0.6911097765, training_acc: 0.500000</code><br>
<code>Epoch [71/1000], loss: 0.6912719607, training_acc: 0.500000</code><br>
<code>Epoch [71/1000], loss: 0.6911540031, training_acc: 0.500000</code><br>
<code>Epoch [71/1000], loss: 0.6911510825, training_acc: 0.500000</code><br>
<code>Epoch [71/1000], loss: 0.6911364794, training_acc: 0.500000</code><br>
<code>Epoch [72/1000], loss: 0.6911281943, training_acc: 0.500000</code><br>
<code>Epoch [72/1000], loss: 0.6911578774, training_acc: 0.500000</code><br>
<code>Epoch [72/1000], loss: 0.6909996867, training_acc: 0.500000</code><br>
<code>Epoch [72/1000], loss: 0.6911479235, training_acc: 0.500000</code><br>
<code>Epoch [72/1000], loss: 0.6910735369, training_acc: 0.500000</code><br>
and my train code is here:</p>
<p><code>import cv2</code><br>
<code>import torch</code><br>
<code>import torch.nn as nn</code><br>
<code>import torch.nn.functional as F</code><br>
<code>from torch.autograd import Variable</code><br>
<code>from model import U_net</code><br>
<code>import visdom</code><br>
<code>from dataset import driveDateset</code><br>
<code>from torch import optim</code><br>
<code>from Dice_loss import DiceLoss</code><br>
<code>from Dice_loss import MulticlassDiceLoss</code><br>
<code>import matplotlib.pylab as plt</code><br>
<code>import numpy as np</code><br>
<code>import time</code><br>
<code>from PIL import Image</code><br>
<code>from pre_processing import my_PreProc</code><br>
<code>from pre_processing import masks_Unet</code></p>
<p><code>if __name__ == '__main__':</code><br>
<code> DATA_DIRECTORY = "F:\\experiment_code\\U-net\\DRIVE\\training"</code><br>
<code> DATA_LIST_PATH = "F:\\experiment_code\\U-net\DRIVE\\training\\images_id.txt"</code><br>
<code>Batch_size = 4</code><br>
<code> epochs = 1000</code><br>
<code>dst = driveDateset(DATA_DIRECTORY, DATA_LIST_PATH)</code><br>
<code> # Initialize model</code><br>
<code> device = torch.device("cuda")</code><br>
<code> model = U_net()</code><br>
<code>model.to(device)</code></p>
<pre><code>#Initial optimizer
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=1e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=(epochs // 9) + 1)
criterion = nn.CrossEntropyLoss() #reduce=False
best_acc, best_epoch =0, 0
global_step = 0
start_time = time.time()
viz = visdom.Visdom()

for epoch in range(epochs):
    scheduler.step(epoch)
    running_corrects = 0
    TP,TN,FN,FP = 0.,0.,0.,0.
    since_epoch = time.time()
    trainloader = torch.utils.data.DataLoader(dst, batch_size=Batch_size,shuffle=True) #,shuffle =True
    for step, data in enumerate(trainloader):
        imgs, labels, _, _ = data
        imgs = imgs.numpy()
        imgs_deal  = my_PreProc(imgs)
        #imgs, labels = imgs.to(device), labels.to(device)
        labels = labels.float()
        labels = labels / 255.0
        labels = labels.reshape(4,1,224,224).numpy()
        mask_train = masks_Unet(labels)
        mask_train = mask_train.reshape(4 * 224 * 224)
        mask_train = torch.from_numpy(mask_train)
        mask_train = mask_train.long()
        imgs_numpy2tensor = torch.from_numpy(imgs_deal)
        imgs_numpy2tensor = imgs_numpy2tensor.to(torch.float32)
        imgs_numpy2tensor,mask_train = imgs_numpy2tensor.to(device),mask_train.to(device)
        model.train()
        outputs = model(imgs_numpy2tensor)   # output  [B*H*W,C]
        loss = criterion(outputs,mask_train)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        viz.line([loss.item()],[global_step], win='loss', update='append',opts=dict(title='train_loss'))
        outputs2 = outputs
        outputs2numpy = outputs2.cpu().detach().numpy().astype(np.float32)
        outputs_compare = np.argmax(outputs2numpy,axis=1)
        mask_train2numpy = mask_train.cpu().numpy().astype(np.float32)
        TP += ((outputs_compare==1) &amp; (mask_train2numpy==1)).sum()
        TN += ((outputs_compare==0) &amp; (mask_train2numpy==1)).sum()
        FN += ((outputs_compare==0) &amp; (mask_train2numpy==1)).sum()
        FP += ((outputs_compare==1) &amp; (mask_train2numpy==0)).sum()
        p = TP / (TP + FP)
        r = TP /(TP + FN)
        F1 = 2 * r * p / (r + p)
        acc = (TP + TN) / (TP + TN + FP + FN)
        time_elapsed_epoch = time.time() - since_epoch
        print('Epoch [%d/%d], loss: %.10f, training_acc: %8f' % (epoch, epochs, loss.item(),acc))
        global_step += 1


time_elapsed = time.time() - start_time
print('Training complate in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
</code></pre>
<p>``</p>

username: ptrblck
cooked: <p>Which line of code is raising this issue?<br>
Could you post the complete error message please?</p>

username: fyy
cooked: <pre><code>TP += ((outputs_compare==1) &amp; (mask_train2numpy==1)).sum()
TN += ((outputs_compare==0) &amp; (mask_train2numpy==1)).sum()
FN += ((outputs_compare==0) &amp; (mask_train2numpy==1)).sum()
FP += ((outputs_compare==1) &amp; (mask_train2numpy==0)).sum()
p = TP / (TP + FP)
r = TP /(TP + FN)
F1 = 2 * r * p / (r + p)
acc = (TP + TN) / (TP + TN + FP + FN)
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  ,thank you, I find my code’s problem ,It is a  RuntimeWarning,not error ,I think may be arithmetic overflow .It should be<br>
<code>TN += ((outputs_compare==0) &amp; (mask_train2numpy==0)).sum()</code></p>