Problem with the training loss of FCN for segmentation

username: Patrice
cooked: <pre><code>                  Dear programmers,
</code></pre>
<p>I am very new to Pytorch  and with very poor programming experience. I have built a network and the training process is as follows:</p>
<p>Epoch_num=5</p>
<p>for e in range(Epoch_num):</p>
<pre><code>train_loss = 0

model= model.train()

for idx, data in tqdm (enumerate(train_loader)):

    optimizer.zero_grad()

    x, y_true = data

    if torch.cuda.is_available():

        x, y_true = x.cuda(), y_true.cuda()

    # forward

    out = model(x)

    out = F.log_softmax(out, dim=1)  # (b, n, h, w)

    loss = criterion(out, y_true)

    # backward

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

    train_loss += loss.item()

    


label_pred = out.max(dim=1)[1].data.cpu()

label_true= y_true.unsqueeze(1)

label_true = label_true .data.cpu()

acc= get_accuracy(label_true,label_pred)

print("Epoch {}/{}, Loss: {:.3f}, Accuracy: {:.3f}".format(e+1,Epoch_num, train_loss, acc))
</code></pre>
<p>The training process is illustrated as follows:</p>
<p>Epoch 1/5, Loss: 765.190, Accuracy: 0.513</p>
<p>0it [00:00, ?it/s]<br>
1it [00:00,  6.00it/s]<br>
2it [00:00,  6.06it/s]<br>
3it [00:00,  6.12it/s]<br>
4it [00:00,  6.11it/s]<br>
5it [00:00,  6.11it/s]<br>
6it [00:00,  6.16it/s]<br>
7it [00:01,  6.11it/s]<br>
8it [00:01,  6.17it/s]<br>
9it [00:01,  6.14it/s]<br>
10it [00:01,  6.14it/s]<br>
11it [00:01,  6.03it/s]<br>
12it [00:01,  5.90it/s]<br>
13it [00:02,  5.96it/s]<br>
14it [00:02,  6.03it/s]<br>
15it [00:02,  6.13it/s]<br>
16it [00:02,  6.05it/s]<br>
17it [00:02,  6.11it/s]<br>
18it [00:02,  6.17it/s]<br>
19it [00:03,  6.13it/s<br>
51it [00:08,  6.13it/s]<br>
52it [00:08,  6.13it/s]<br>
53it [00:08,  6.09it/s]<br>
54it [00:08,  6.04it/s]<br>
55it [00:09,  6.11it/s]<br>
56it [00:09,  6.10it/s]<br>
57it [00:09,  6.16it/s]<br>
58it [00:09,  5.97it/s]<br>
59it [00:09,  6.02it/s]<br>
60it [00:09,  6.08it/s]<br>
61it [00:09,  6.10it/s]<br>
62it [00:10,  6.06it/s]<br>
63it [00:10,  6.02it/s]<br>
64it [00:10,  6.13it/s]<br>
65it [00:10,  6.11it/s]<br>
66it [00:10,  6.11it/s]<br>
67it [00:10,  6.15it/s]<br>
68it [00:11,  6.14it/s]<br>
69it [00:11,  6.15it/s]<br>
70it [00:11,  6.19it/s]<br>
71it [00:11,  6.11it/s]<br>
72it [00:11,  6.11it/s]<br>
73it [00:11,  6.14it/s]<br>
74it [00:12,  6.14it/s]<br>
75it [00:12,  6.04it/s]<br>
76it [00:12,  6.06it/s]<br>
77it [00:12,  6.09it/s</p>
<p>1101it [03:00,  6.08it/s]<br>
1102it [03:00,  6.12it/s]<br>
1103it [03:00,  6.20it/s]<br>
1104it [03:00,  6.22it/s]<br>
Epoch 2/5, Loss: 765.112, Accuracy: 0.514</p>
<p>1103it [02:54,  6.36it/s]<br>
1104it [02:55,  6.29it/s]<br>
Epoch 3/5, Loss: 764.840, Accuracy: 0.535</p>
<p>1104it [03:00,  6.10it/s]<br>
Epoch 4/5, Loss: 761.322, Accuracy: 0.704</p>
<p>It can be seen that the loss does decrease while the accuracy increases. However, the loss is still very high. Please, could you help me check what is wrong with my implementation? Moreover, how can I modify the codes in such a way that not every mini-batch will be displayed during training?</p>
<p>Thank you very much for your time and guidance</p>

username: ptrblck
cooked: <p>Based on the usage of <code>F.log_softmax</code> I assume you are using <code>nn.NLLLoss</code> as your criterion?<br>
If so, did you pass any arguments as the <code>reduction</code> or are you using the default settings?</p>
<p>To remove the printing you might just want to remove the <code>tqdm</code> call. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Patrice
cooked: <p>I am very grateful for your reply sir.<br>
I was using the default setting. After your suggestion, I went and read the documentation. However, I still can not figure it out.</p>
<p>when I set reduction=‘sum’, the loss is ‘Nan’<br>
when I set reduction=‘none’, it gives the following error ‘RuntimeError: grad can be implicitly created only for scalar outputs’<br>
I have searched online, but could not find any suitable solutions</p>
<p>As for the printing, removing ‘tqdm’ call solved the problem. However, is there any way by which I could print the loss after a certain number of batches?</p>
<p>Thank you for your time and guidance</p>

username: ptrblck
cooked: <p>How many classes are you using?<br>
The loss value seem to be quite high for a reasonable accuracy, but might of course still be valid.</p>
<p>If you don’t reduce the loss, you would have to pass the gradient into <code>backward</code> with the same shape as your output.</p>
<p>The usual approach of printing some statistics is to use a condition inside your training loop as seen e.g. <a href="https://github.com/pytorch/examples/blob/e9e76722dad4f4569651a8d67ca1d10607db58f9/mnist/main.py#L45-L48">here</a>.</p>

username: Patrice
cooked: <p>My task is a  binary segmentation (foreground and background)</p>
<p>If you don’t reduce the loss, you would have to pass the gradient into  <code>backward</code>  with the same shape as your output. Please, could you help me with an example?</p>
<p>I have read the work you recommended and I have modified my codes as follows:</p>
<p>Epoch_num=5</p>
<p>for e in range(Epoch_num):</p>
<pre><code>print('Epoch', e+1)

train_loss = 0

model= model.train()



for idx, data in enumerate(train_loader):

#for idx, data in tqdm (enumerate(train_loader)):

    optimizer.zero_grad()

    x, y_true = data

    if torch.cuda.is_available():

        x, y_true = x.cuda(), y_true.cuda()

    # forward

    out = model(x)

    out = F.log_softmax(out, dim=1)  # (b, n, h, w)

    #loss = criterion(out[0,0,:,:], y_true[0,:,:].type_as(out))

    loss = criterion(out, y_true)

    # backward

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

    train_loss += loss.item()

    if idx % 100 == 0 and idx != 0:

        print('[{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(

             idx , len(train_loader.dataset),

            100. * idx / len(train_loader), train_loss/ idx))

       

label_pred = out.max(dim=1)[1].data.cpu()

label_true= y_true.unsqueeze(1)

label_true = label_true .data.cpu()

acc= get_accuracy(label_true,label_pred)

print("Epoch {}/{}, Loss: {:.5f}, Accuracy: {:.5f}".format(e+1,Epoch_num, train_loss/len(train_loader.dataset), acc))
</code></pre>
<p>As it can be seen, I have divided the total training loss by the number of samples in the training data. Please, kindly check and see whether doing so is logical or not.</p>
<p>Some statistics of the training process are given below</p>
<p>Epoch 1</p>
<p>[100/1104 (9%)] Loss: 0.700105</p>
<p>[200/1104 (18%)]    Loss: 0.696637</p>
<p>[300/1104 (27%)]    Loss: 0.695482</p>
<p>[400/1104 (36%)]    Loss: 0.694904</p>
<p>[500/1104 (45%)]    Loss: 0.694557</p>
<p>[600/1104 (54%)]    Loss: 0.694326</p>
<p>[700/1104 (63%)]    Loss: 0.694161</p>
<p>[800/1104 (72%)]    Loss: 0.694037</p>
<p>[900/1104 (82%)]    Loss: 0.693941</p>
<p>[1000/1104 (91%)]   Loss: 0.693863</p>
<p>[1100/1104 (100%)]  Loss: 0.693800</p>
<p>Epoch 1/5, Loss: 0.69317, Accuracy: 0.48804</p>
<p>Epoch 2</p>
<p>[100/1104 (9%)] Loss: 0.700100</p>
<p>[200/1104 (18%)]    Loss: 0.696634</p>
<p>[300/1104 (27%)]    Loss: 0.695478</p>
<p>[400/1104 (36%)]    Loss: 0.694899</p>
<p>[500/1104 (45%)]    Loss: 0.694552</p>
<p>[600/1104 (54%)]    Loss: 0.694321</p>
<p>[700/1104 (63%)]    Loss: 0.694156</p>
<p>[800/1104 (72%)]    Loss: 0.694032</p>
<p>[900/1104 (82%)]    Loss: 0.693935</p>
<p>[1000/1104 (91%)]   Loss: 0.693858</p>
<p>[1100/1104 (100%)]  Loss: 0.693795</p>
<p>Epoch 2/5, Loss: 0.69317, Accuracy: 0.48364</p>
<p>Epoch 3</p>
<p>[100/1104 (9%)] Loss: 0.700095</p>
<p>[200/1104 (18%)]    Loss: 0.696631</p>
<p>[300/1104 (27%)]    Loss: 0.695475</p>
<p>[400/1104 (36%)]    Loss: 0.694896</p>
<p>[500/1104 (45%)]    Loss: 0.694549</p>
<p>[600/1104 (54%)]    Loss: 0.694317</p>
<p>[700/1104 (63%)]    Loss: 0.694152</p>
<p>[800/1104 (72%)]    Loss: 0.694028</p>
<p>[900/1104 (82%)]    Loss: 0.693932</p>
<p>[1000/1104 (91%)]   Loss: 0.693855</p>
<p>[1100/1104 (100%)]  Loss: 0.693791</p>
<p>Epoch 3/5, Loss: 0.69316, Accuracy: 0.49097</p>
<p>Epoch 4</p>
<p>[100/1104 (9%)] Loss: 0.700089</p>
<p>[200/1104 (18%)]    Loss: 0.696625</p>
<p>[300/1104 (27%)]    Loss: 0.695470</p>
<p>[400/1104 (36%)]    Loss: 0.694892</p>
<p>[500/1104 (45%)]    Loss: 0.694545</p>
<p>[600/1104 (54%)]    Loss: 0.694314</p>
<p>[700/1104 (63%)]    Loss: 0.694149</p>
<p>[800/1104 (72%)]    Loss: 0.694025</p>
<p>[900/1104 (82%)]    Loss: 0.693928</p>
<p>[1000/1104 (91%)]   Loss: 0.693851</p>
<p>[1100/1104 (100%)]  Loss: 0.693788</p>
<p>Epoch 4/5, Loss: 0.69316, Accuracy: 0.50049</p>