Custom Loss Function Problem

username: Mr.Bank
cooked: <p>Hi,<br>
Is there any code line in my loss function to break backpropagation? Because, after started training, loss value plays around the started value, not changed dramatically.<br>
I could not find why.<br>
Thanks in advance.</p>
<p>Train Loses in 10 times epoch:<br>
Train loss: 66.125526<br>
Train loss: 69.031129<br>
Train loss: 66.471873<br>
Train loss: 74.184909<br>
Train loss: 62.160489<br>
Train loss: 63.237526<br>
Train loss: 63.310347<br>
Train loss: 62.977487<br>
Train loss: 71.276782<br>
Train loss: 64.684782</p>
<pre><code class="lang-auto">class pairWiseLoss(nn.Module):    
    
    def __init__(self,lambdaValue,lenghtOfHashCode):
        super(pairWiseLoss, self).__init__()
        self.lambdaValue = lambdaValue
        self.l = lenghtOfHashCode
        self.m = lenghtOfHashCode
    
    
    def forward(self, binary1, labels1, binary2, labels2):

        
        similarity = torch.diagonal(torch.mm(labels1, labels2.t()))
        maskCommonLabel = similarity.gt(0.0)
        maskNoCommonLabel = similarity.eq(0.0)
        
        tc = torch.exp(-similarity) * self.lambdaValue * 4 * self.l
        hammingDistance = torch.diagonal( torch.cdist(binary1,binary2, p = 0) )
        
        loss = torch.sum(torch.masked_select(0.5 * F.relu(hammingDistance - tc),maskCommonLabel  )  ) +  torch.sum(torch.masked_select(0.5 * F.relu( self.m - hammingDistance), maskNoCommonLabel ) )
                
        return loss 
</code></pre>

username: ptrblck
cooked: <p>I can’t see any line of code, which would detach a tensor from the graph.<br>
If you are concerned about detaching, you could check the <code>.grad</code> attribute of all parameters after the <code>backward</code> call. If they contain valid values, your graph wasn’t detached, and I would recommend to try to overfit a small data sample as a quick test.</p>

username: Mr.Bank
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the reply.</p>
<pre><code class="lang-auto">        loss.backward()
        print('loss Grad:' , loss.grad)
</code></pre>
<p>It returns None. Is it normal?<br>
I checked it with a predefined loss function. Training loss value are decreasing as planned but loss.grad is still None.</p>
<p>Should I write .grad for the network model layers? I could not understand what all parameters mean.</p>
<p>Also, my dataset is small. 500 training, 250 validation and 250 test samples.<br>
When I use the big dataset, the loss value still does not decrease.</p>

username: ptrblck
cooked: <p>The <code>grad</code> attribute will be retained for leaf variables by default.<br>
If you want to print them for the <code>loss</code>, you would need to call <code>loss.retain_grad()</code> before calling <code>loss.baclward()</code>.<br>
However, note that this gradient will be <code>1.</code> by default, if you didn’t pass any manual gradient argument to <code>loss.backward(gradient=)</code>.<br>
Here is a small example:</p>
<pre><code class="lang-python">model = models.resnet18()
x = torch.randn(1, 3, 224, 224)
target = torch.zeros(1).long()

criterion = nn.CrossEntropyLoss()

out = model(x)
loss = criterion(out, target)
loss.retain_grad() # use this to print the grad
loss.backward()

print(loss.grad)
&gt; tensor(1.)
</code></pre>
<p>To print the gradient of all parameters, you could use this code snippet after calling <code>backward</code>:</p>
<pre><code class="lang-python"># print grads of all parameters
for name, param in model.named_parameters():
    print(name, param.grad.abs().max())
</code></pre>

username: Mr.Bank
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> hi again,<br>
as you said, loss.grad returns 1. However, the gradient of the parameters seems there are some problems.<br>
All parameters have the same weight and bias regardless of the epoch except FC.bias tensor.<br>
At least, they seem same but I used your code that mentioned here:</p><aside class="quote quote-modified" data-post="2" data-topic="21408">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/weights-not-properly-updating/21408/2">Weights not properly updating</a> 
  </div>
  <blockquote>
    To see if your weight got updated you could use the following scheme: 
# Save init values
old_state_dict = {}
for key in model.state_dict():
old_state_dict[key] = model.state_dict()[key].clone()

# Your training procedure
...

# Save new params
new_state_dict = {}
for key in model.state_dict():
    new_state_dict[key] = model.state_dict()[key].clone()

# Compare params
for key in old_state_dict:
    if not (old_state_dict[key] == new_state_dict[key]).all():
        print('Diff in {}'.format(key)…
  </blockquote>
</aside>
<p>
Many parameters have been printed so it means there are some changes between new and old state but it is so small for example:</p>
<pre><code class="lang-auto"> old_state_dict['encoder.1.weight']
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
</code></pre>
<pre><code class="lang-auto">
new_state_dict['encoder.1.weight']
tensor([0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997,
        0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997,
        0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997,
        0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997,
        0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997,
        0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997,
        0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997, 0.9997,
        0.9997])
</code></pre>
<p>Epoch 0:<br>
FC.bias tensor(8.7036e-05, device=‘cuda:0’)<br>
FC.bias tensor(8.7035e-05, device=‘cuda:0’)<br>
FC.bias tensor(8.7035e-05, device=‘cuda:0’)<br>
Epoch 1:<br>
FC.bias tensor(8.7034e-05, device=‘cuda:0’)<br>
FC.bias tensor(8.7032e-05, device=‘cuda:0’)<br>
FC.bias tensor(8.7031e-05, device=‘cuda:0’)<br>
Epoch 2:<br>
FC.bias tensor(8.7029e-05, device=‘cuda:0’)<br>
FC.bias tensor(8.7028e-05, device=‘cuda:0’)<br>
FC.bias tensor(8.7026e-05, device=‘cuda:0’)</p>
<p>So are these updates enough? Can it be the main reason of not decreasing loss value?</p>
<p>My network model:</p>
<pre><code class="lang-auto">class ResNet50PairWise(nn.Module):
    def __init__(self, bits = 16):
        super().__init__()

        resnet = models.resnet50(pretrained=False)

        self.conv1 = nn.Conv2d(12, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
        self.encoder = nn.Sequential(
            self.conv1,
            resnet.bn1,
            resnet.relu,
            resnet.maxpool,
            resnet.layer1,
            resnet.layer2,
            resnet.layer3,
            resnet.layer4,
            resnet.avgpool
        )
        self.FC = nn.Linear(2048, bits)


        self.apply(weights_init_kaiming)
        self.apply(fc_init_weights)

    def forward(self, x):
        x = self.encoder(x)
        x = x.view(x.size(0), -1)

        logits = self.FC(x)
        sign = torch.sign(logits)
        binary_out = torch.relu(sign)

        return binary_out```</code></pre>

username: ptrblck
cooked: <p>It might be the reason and as explained in the other topic, the <code>sign</code> method could kill the gradients.<br>
You could try to increase the learning rate and compare how large the updates would get (you can of course also increase the learning rate for a specific parameter set only).<br>
Alternatively, you could try to use smooth approximations of the <code>sign</code> function, which wouldn’t yield a zero gradient.</p>