Longer training epochs when calculating the loss by hand

username: rcshubhadeep
cooked: <p>Hello All,</p>
<p>This is my first post and I am sorry if either I posting in a wrong place or this post is too trivial. But I can’t really figure out what is going wrong here. I will come to the point. I have a tensor looking like this</p>
<pre><code class="lang-auto">X = torch.ones((6, 8))
X[:, 2:6] = 0
X

tensor([[1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.],
        [1., 1., 0., 0., 0., 0., 1., 1.]])
</code></pre>
<p>And I have a Kernel looking like this (Simple vertical edge detection kernel)</p>
<p><code>K = torch.tensor([[1, -1]])</code></p>
<p>When I apply this kernel on the tensor I get the desired result</p>
<pre><code class="lang-auto">Y
tensor([[ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.],
        [ 0.,  1.,  0.,  0.,  0., -1.,  0.]])
</code></pre>
<p>Now I want to learn this kernel. Here is what my code looks like</p>
<pre><code class="lang-auto">conv = nn.Conv2d(1, 1, kernel_size=(1, 2), bias=False)

X = X.reshape(1, 1, 6, 8)
Y = Y.reshape(1, 1, 6, 7)

for i in range(10):
    Y_hat = conv(X)
    loss = (Y_hat - Y) ** 2
    conv.zero_grad()
    loss.sum().backward()
    
    conv.weight.data[:] -= 3e-2 * conv.weight.grad
    if i % 2 == 0:
        print(f"Loss at {i} epoch is {loss.sum()}")
</code></pre>
<p>When I run this algorithm for 10 epochs (as shown in the code above), and then print out the weights of the kernel I get this</p>
<pre><code class="lang-auto">Parameter containing:
tensor([[[[ 0.9466, -1.0235]]]], requires_grad=True)
</code></pre>
<p>Which is very close to the actual values.</p>
<p>However, when I change the loss calculation my nn.MSELoss Like so -</p>
<pre><code class="lang-auto">...
loss = nn.MSELoss()(Y_hat, Y)
...
loss.backward()
</code></pre>
<p>Training for only 10 epochs does not approximate at all (something like this - <code>tensor([[[[-0.0427,  0.2747]]]], requires_grad=True)</code> )</p>
<p>And when I train it for much longer (let’s say 150 epochs or so) then it makes a bit better approximation</p>
<p>Like so - <code>tensor([[[[ 0.7467, -0.7456]]]], requires_grad=True)</code></p>
<p>I am curious to know why this huge difference of epochs are there just because I changed the loss calculation slightly.</p>
<p>Thanks for pointing me to any good resources.</p>
<p>Regards,<br>
Shubhadeep</p>

username: ptrblck
cooked: <p>Your manual approach is not reducing the loss in the first call:</p>
<pre><code class="lang-python">loss = (Y_hat - Y) ** 2
</code></pre>
<p>so <code>loss</code> will have the shape <code>[1, 1, 6, 7]</code> and later you will <code>sum</code> this loss.</p>
<p>However, <code>nn.MSELoss</code> uses the <code>reduction='mean'</code> by default, which would yield a lower loss value and thus also gradients with a smaller magnitude.<br>
If you use <code>loss = nn.MSELoss(reduction='sum')(Y_hat, Y)</code>, you would get a similar result.</p>

username: rcshubhadeep
cooked: <p>Thank you very much. Now that I think about it, it is trivial. But I was not able to get before. Your answer clears a doubt. Thanks.</p>
<p>However, here is a curious question, as <code>sum</code> reduction will produce a larger gradient and thus the converging can be faster, why the choice of <code>mean</code> by default? We can always reduce the learning rate in order to regularize the steps if they are too big and we are running into not finding the minima. What is your take on this? Will be curious to know.</p>
<p>Have a good day and thanks again.</p>
<p>Best</p>

username: ptrblck
cooked: <p>The <code>reduction='mean'</code> default is chosen, as it will create gradients which do not depend on the batch size.<br>
E.g. the <code>DataLoader</code> could return a smaller sample size for the last batch, of the length of the dataset is divisible by the batch size with a remainder. This would scale down the gradients, if you are using <code>reduction='sum'</code>. The same applies of course if you are manually changing the batch size somehow.</p>
<p>Also, if you are using a weighted loss, <code>recution='mean'</code> will use the applied weights to normalize the reduced loss (as given in the <a href="https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html">docs</a>). This will not be the case for <code>reduction='sum'</code> (the weighted loss will just be summed), which might increase or decrease the loss depending which classes (and thus weights) are found in the current batch.</p>

username: rcshubhadeep
cooked: <p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: rcshubhadeep
cooked: <p>Thank you very much. It is clear now.</p>
<p>Thanks for the clarification.</p>
<p>Best regards</p>