My model even can't overfit two image and the loss do not decrease!

ALanLee: <p>Epoch [1/5], Iter [1/51200] TotalLoss: 14.3430 Loss1: 3.4093 Loss2: 3.7797 Loss3: 3.4102 Loss3: 3.7438<br>
Epoch [1/5], Iter [2/51200] TotalLoss: 9.1436 Loss1: 2.0375 Loss2: 2.9503 Loss3: 2.7624 Loss3: 1.3935<br>
Epoch [1/5], Iter [3/51200] TotalLoss: 12.7019 Loss1: 3.1669 Loss2: 3.1706 Loss3: 3.1461 Loss3: 3.2183<br>
Epoch [1/5], Iter [4/51200] TotalLoss: 199.0285 Loss1: 45.2067 Loss2: 60.3769 Loss3: 52.5529 Loss3: 40.8920<br>
Epoch [1/5], Iter [5/51200] TotalLoss: 13.5216 Loss1: 3.3813 Loss2: 3.3809 Loss3: 3.3761 Loss3: 3.3833<br>
Epoch [1/5], Iter [6/51200] TotalLoss: 12.3115 Loss1: 3.0906 Loss2: 3.0759 Loss3: 3.0606 Loss3: 3.0844<br>
Epoch [1/5], Iter [7/51200] TotalLoss: 9.6911 Loss1: 2.4250 Loss2: 2.0567 Loss3: 2.1717 Loss3: 3.0377<br>
Epoch [1/5], Iter [8/51200] TotalLoss: 11.4812 Loss1: 2.8772 Loss2: 2.8731 Loss3: 2.8619 Loss3: 2.8690<br>
Epoch [1/5], Iter [9/51200] TotalLoss: 7.7518 Loss1: 1.9965 Loss2: 1.8900 Loss3: 1.9414 Loss3: 1.9240<br>
Epoch [1/5], Iter [10/51200] TotalLoss: 5.8163 Loss1: 1.3725 Loss2: 1.4375 Loss3: 1.5680 Loss3: 1.4383<br>
Epoch [1/5], Iter [11/51200] TotalLoss: 5.2633 Loss1: 1.3306 Loss2: 1.3322 Loss3: 1.2640 Loss3: 1.3365<br>
Epoch [1/5], Iter [12/51200] TotalLoss: 3.2123 Loss1: 0.8192 Loss2: 0.7681 Loss3: 0.7401 Loss3: 0.8849<br>
Epoch [1/5], Iter [13/51200] TotalLoss: 2.6675 Loss1: 0.6967 Loss2: 0.6703 Loss3: 0.6426 Loss3: 0.6579<br>
Epoch [1/5], Iter [14/51200] TotalLoss: 2.8774 Loss1: 0.7332 Loss2: 0.7385 Loss3: 0.6811 Loss3: 0.7245<br>
Epoch [1/5], Iter [15/51200] TotalLoss: 2.8546 Loss1: 0.7366 Loss2: 0.7264 Loss3: 0.7202 Loss3: 0.6713<br>
Epoch [1/5], Iter [16/51200] TotalLoss: 2.7758 Loss1: 0.6806 Loss2: 0.6549 Loss3: 0.7345 Loss3: 0.7058<br>
Epoch [1/5], Iter [17/51200] TotalLoss: 2.8139 Loss1: 0.7092 Loss2: 0.6574 Loss3: 0.7800 Loss3: 0.6674<br>
Epoch [1/5], Iter [18/51200] TotalLoss: 2.7282 Loss1: 0.6225 Loss2: 0.6997 Loss3: 0.7237 Loss3: 0.6823<br>
Epoch [1/5], Iter [19/51200] TotalLoss: 2.8259 Loss1: 0.7428 Loss2: 0.7204 Loss3: 0.6845 Loss3: 0.6783<br>
Epoch [1/5], Iter [20/51200] TotalLoss: 2.5689 Loss1: 0.6701 Loss2: 0.6525 Loss3: 0.6837 Loss3: 0.5626<br>
Epoch [1/5], Iter [21/51200] TotalLoss: 2.7335 Loss1: 0.6944 Loss2: 0.7051 Loss3: 0.6600 Loss3: 0.6740<br>
Epoch [1/5], Iter [22/51200] TotalLoss: 2.8239 Loss1: 0.6977 Loss2: 0.7478 Loss3: 0.6584 Loss3: 0.7199<br>
Epoch [1/5], Iter [23/51200] TotalLoss: 2.7279 Loss1: 0.7237 Loss2: 0.6032 Loss3: 0.6741 Loss3: 0.7270<br>
Epoch [1/5], Iter [24/51200] TotalLoss: 2.6124 Loss1: 0.6750 Loss2: 0.6018 Loss3: 0.6210 Loss3: 0.7146<br>
Epoch [1/5], Iter [25/51200] TotalLoss: 2.9644 Loss1: 0.7071 Loss2: 0.6792 Loss3: 0.7112 Loss3: 0.8669<br>
Epoch [1/5], Iter [26/51200] TotalLoss: 3.0921 Loss1: 0.6923 Loss2: 0.7369 Loss3: 0.6726 Loss3: 0.9902<br>
Epoch [1/5], Iter [27/51200] TotalLoss: 3.1947 Loss1: 0.7637 Loss2: 0.8234 Loss3: 0.7893 Loss3: 0.8183<br>
Epoch [1/5], Iter [28/51200] TotalLoss: 2.9257 Loss1: 0.6981 Loss2: 0.7647 Loss3: 0.6602 Loss3: 0.8028<br>
my network is vgg like artitecture and the task is to recognize captcha<br>
it’s a multi-label task by using cnn.<br>
but it didn’t work, it cannot  even fit two image by training a lot of steps…<img src="//discuss.pytorch.org/images/emoji/apple/expressionless.png?v=5" title=":expressionless:" class="emoji" alt=":expressionless:"><img src="//discuss.pytorch.org/images/emoji/apple/expressionless.png?v=5" title=":expressionless:" class="emoji" alt=":expressionless:"><img src="//discuss.pytorch.org/images/emoji/apple/expressionless.png?v=5" title=":expressionless:" class="emoji" alt=":expressionless:"></p>

Alessandro_De_Micco: <p>How is defined your Loss function?</p>

spacemeerkat: <p>I have a similar problem where I have a simple model which won’t overfit despite not using regularising techniques like dropout.</p>
<p>My model looks as follows:</p>
<pre><code class="lang-auto">Net(
  (feature_extractor): Sequential(
    (0): Conv2d(1, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): Conv2d(64, 128, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (3): ReLU()
    (4): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(128, 256, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (7): ReLU()
    (8): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=65536, out_features=256, bias=True)
    (1): ReLU()
    (2): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (3): Linear(in_features=256, out_features=1, bias=True)
  )
)

</code></pre>
<p>and my training loop looks as follows:</p>
<pre><code class="lang-auto">lr = np.linspace(0.01,1./(1000**2),25)

torch.backends.cudnn.benchmark = True
torch.backends.cudnn.fastest = True

start = time.time()
model.train()
train_loss = []
train_accu = []
i = 0
for epoch in range(25):
    optimizer = optim.Adam(model.parameters(), lr=lr[epoch])
    for image, label in train_loader:
        data, target = image.float().cuda(), label.float().cuda()
        optimizer.zero_grad()
        output = model(data)
        loss = F.mse_loss(output, target.view(output.size()))  
        loss.backward()    # calc gradients
        train_loss.append(loss.data[0]) # Calculating the loss
        optimizer.step()   # update gradients
        prediction = torch.floor(output).int()  

        accuracy = (prediction.eq(target.view(output.size()).int()).sum().float()/batch_size)*100
        train_accu.append(accuracy)
        if i % 10 == 0:
            print('Epoch:',str(epoch),'Train Step: {}\tLoss: {:.3f}\tAccuracy: {:.3f}'.format(i, loss.data[0], accuracy))
            #print('TARGET:',target)
            #print('OUTPUT:',output.view(target.size()))
        i += 1
end = time.time()
print('TRAIN TIME:')
print('%.2gs'%(end-start))
</code></pre>
<p>I would have thought that after 10 iterations the model would have at least started to increase in accuracy towards overfitting but so far I’m just getting a bouncing of accuracy below ~30%.</p>
<p>Loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4f8889fe5a66bdc233c15012c85d74acbd312f67.png" alt="loss" data-base62-sha1="blAfJeJWUBraIxLWQryfFyMchsX" width="483" height="322"><br>
Accuracy:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/ce31dcdabac66980d196336243bf496458639dc3.png" alt="accuracy2" data-base62-sha1="tq5bJB4dnUDi2DTAM0nVn3SnzQn" width="483" height="336"></p>
<p>TL,DR:</p>
<p>mse_loss<br>
Adam optimizer<br>
Learning rate decay<br>
No dropout (but do include batch norm)<br>
Not overfitting after 10 epochs.</p>
<p>Thanks in advance for any guidance on this issue!</p>

ptrblck: <p>Could you try to change the learning rate using one optimizer and changing its <code>param_groups[0]['lr']</code> or alternatively <a href="https://pytorch.org/docs/master/optim.html#how-to-adjust-learning-rate">torch.optim.lr_scheduler</a>?<br>
Currently you are re-initializing it in each epoch, thus your optimizer will lose its accumulated momentum etc.<br>
Maybe it won’t have a big effect, but it’s worth a first shot!</p>

spacemeerkat: <p>Thanks for the reply, I’m now using:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters(), lr=0.01)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.1)
</code></pre>
<p>and for training I simply include the line:</p>
<p><code>scheduler.step()</code></p>
<p>under the <code>for epoch in range(25):</code> loop.</p>
<p>I will update once it has undergone all it’s iterations…</p>
<p>UPDATE:</p>
<p>That hasn’t had an affect on improving training accuracy unfortunately, but thanks for pointed out that bug <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=5" title=":grin:" class="emoji" alt=":grin:"></p>

Sherlock_c787: <p>Have you solved the problem? I meet the same problem, and i tried to change the network structure, learning rate and training strategy, it still dosen’t work.<br>
my loss function is shown in the following picture:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a22975ffb7a5ad9ebcb091481404aaf53dae6ee8.png" data-download-href="https://discuss.pytorch.org/uploads/default/a22975ffb7a5ad9ebcb091481404aaf53dae6ee8" title="%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a22975ffb7a5ad9ebcb091481404aaf53dae6ee8.png" alt="%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0" data-base62-sha1="n8yb9lRPoa3YUy07BiSgRtzhBcA" width="690" height="143" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0</span><span class="informations">705×147 22.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thanks in advance!</p>

Deeply: <p>I could be wrong, but I do not think using the term ‘overfit’ is legit, both in your question and the post.</p>