Very small learning rate needed for convergence

username: tpostadj
cooked: <p>I am training a custom patch-based network (4 layers) and I realized that, unless I set the lr to 0.0000001 to start converging somehow.<br>
I feel like something is wrong with such a tuning.<br>
However, if I run the training with lr = 0.01, loss will get huge, like several billions and eventually NaN.</p>
<p>Can this be related to initialisation ?</p>

username: ptrblck
cooked: <p>It could be related to the weight init or other hyper-parameters.<br>
How do you initialize your model?<br>
Which optimizer are you using? Did you change the momentum (if available)?</p>

username: tpostadj
cooked: <p>Weight initialisation is done through Xavier’s approach :<br>
m.weight.data.normal_(0, math.sqrt(2. / n)), for each conv module m in the network<br>
m.weight.data.normal_(0, 0.01), for the fc layer on top of the net.</p>
<p>As for momentum, I used the commonly 0.9 value although I do not know it that suits the learning rate I have since this one is really low.<br>
I have to admit I am not really familiar with the relation between learning rate and momentum since both of them seem to impact the weight update on different aspects.</p>
<p>I was looking for more documentation about these two hyper-parameters when I came across weight decay. I understood that it aims at reducing the magnitude increase of the weights. Is that correct ?</p>
<p>Not sure how to tune momentum (and weight decay ?) so as to get a decent learning rate.</p>

username: ptrblck
cooked: <p>Ok, thanks for the update!<br>
Could you post the code calculating the loss, the optimizer and if possible the model?<br>
Maybe your gradients are really high somehow.</p>
<p>Usually a momentum of <code>0.9</code> should work fine.</p>
<p>Yes, weight decay penalizes the weight magnitude, forcing it to get lower values.</p>

username: tpostadj
cooked: <p>Thank you for such quick and helpful answers, really nice community !</p>
<p>Model (just the initilisation) --&gt; comes from torchvision:</p>
<pre><code>    def _initialize_weights(self):
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
            m.weight.data.normal_(0, math.sqrt(2. / n))
            if m.bias is not None:
                m.bias.data.zero_()
        elif isinstance(m, nn.BatchNorm2d):
            m.weight.data.fill_(1)
            m.bias.data.zero_()
        elif isinstance(m, nn.Linear):
            m.weight.data.normal_(0, 0.01)
            m.bias.data.zero_()
</code></pre>
<p>Optimizer:</p>
<pre><code>optimizer = optim.SGD(net.parameters(),
                  lr=0.00000001,
                  momentum=0.9,
                  weight_decay=0.0005)  
</code></pre>
<p>Loss computation:</p>
<pre><code>def runEpoch(self):
    loss_list = np.zeros(100000)
    for it, batch in enumerate(tqdm(self.data_loader)):
        data = Variable(batch['image'])
        target = Variable(batch['class_code'])
        # forward
        if self.mode == 'cuda':
            data = data.cuda()
            target = target.long().cuda()
        output = self.model.forward(data)
        loss = self.criterion(output.float(), target)
        loss.backward()
        self.optimizer.step()
        loss_list[it] = loss.item()
    self.avg_loss = np.mean(loss_list[np.nonzero(loss_list)])
</code></pre>
<p>Also, note that I am using 4-band 8-bit encoded images as input.</p>

username: ptrblck
cooked: <p>From skimming your code, it looks like you are not zeroing out the gradients after the weight update.<br>
In this case the gradients get accumulated and the weight updates will be quite useless.</p>
<p>Add this line into your for loop and run it again:</p>
<pre><code>self.optimizer.zero_grad()
</code></pre>
<p>It is also recommended to call the model directly to compute the forward pass instead of <code>model.forward()</code>.<br>
If you use <code>model.forward()</code> the hooks won’t be called, which might be unimportant in your current code, but might lead to errors in the future. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Also, you could update to the latest stable release (<code>0.4.0</code>). You can find the install instructions on the <a href="https://pytorch.org/">website</a>.</p>

username: tpostadj
cooked: <p>Thank you very much, just tried it and it improves the performance by almost 8%!<br>
I am also able to use a much more reasonable learning rate (0.001, i will try with learning rate decay as well) !<br>
Why is it so important to zero gradients ? Is that because it would otherwise compute gradients and add them addition to those from the previous iteration (sorry… not sure of the meaning of <em>accumulating</em> here) ?</p>
<p>Can you imagine a situation were you would not do such a thing ? From what I have seen, it could be implicit when using optimizer.step()</p>
<p>I am already using 0.4.0 release <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"><br>
Thank you again !</p>

username: ptrblck
cooked: <p>Yes, your explanation is right. The gradients from each backward pass would be summed to the previous gradients.</p>
<p>You could use it to artificially increase your batch size.<br>
If you don’t have enough GPU memory, but need a larger batch size, you could sum the gradients for several smaller batches, average them, and finally perform the optimization step.</p>
<p>Ah ok, then you don’t need the <code>Variable</code> wrapper anymore, since they were merged with <code>tensor</code>s. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

username: monster
cooked: <p>I am using Adamax optimizer with a learning rate = 0.001 and momentum = 0.9, It is working fine for less number of classes (1-50 classes) but for a large number of classes (200-300) I am getting low accuracy and high loss. what should I do in this case?<br>
Thanks in advance</p>