Custom Loss Function Does not converge

username: Mr.Bank
cooked: <p>Hi,<br>
I am new in Pytorch and try to implement a custom loss function which is mentioned in a paper, Deep Multi-Similarity Hashing for Multi-label Image Retrieval.<br>
However, the loss value from the loss function does not converge to smaller values whether used big epoch numbers or not. It plays around the started value.<br>
I checked weight values and they are changing so I think backward mechanisms work.<br>
So where is the problem?</p>
<p>The loss function:</p>
<pre><code class="lang-auto">class pairWiseLoss(nn.Module):    
    def forward(self, binaryCodes, labels):

        lambdaValue = 0.62
        
        numSample = len(binaryCodes)
        halfNumSample = numSample // 2
    
        
        similarity = torch.diagonal(torch.mm(labels[:halfNumSample], labels[halfNumSample:].t()))
        maskCommonLabel = similarity.gt(0.0)
        maskNoCommonLabel = similarity.eq(0.0)
        
        tc = torch.exp(-similarity) * lambdaValue * 4 * args.bits
        m = 2 * args.bits
        hammingDistance = torch.diagonal( torch.cdist(binaryCodes[:halfNumSample],binaryCodes[halfNumSample:], p = 0) )
        zeros = torch.zeros_like(hammingDistance)
        
        loss = torch.sum(torch.masked_select(0.5 * torch.max(hammingDistance - tc , zeros ),maskCommonLabel  )  ) +  torch.sum(torch.masked_select(0.5 * torch.max( m - hammingDistance, zeros), maskNoCommonLabel ) )
        
        return loss   
</code></pre>
<p>The training block:</p>
<pre><code class="lang-auto">def train(trainloader, model, optimizer, epoch, use_cuda, train_writer,gpuDisabled,customLoss):

    label_train = []
    name_train = []
    generated_codes = []
    
    
    lossTracker = MetricTracker()
    
    model.train()
    
    for idx, data in enumerate(tqdm(trainloader, desc="training")):
        
        numSample = data["bands10"].size(0)

        if gpuDisabled :
            bands = torch.cat((data["bands10"], data["bands20"],data["bands60"]), dim=1).to(torch.device("cpu"))
            labels = data["label"].to(torch.device("cpu")) 
        else:            
            bands = torch.cat((data["bands10"], data["bands20"],data["bands60"]), dim=1).to(torch.device("cuda"))
            labels = data["label"].to(torch.device("cuda"))
        

        optimizer.zero_grad()
        logits = model(bands)        
        loss = customLoss(logits, labels)

        loss.backward()
        optimizer.step()

        lossTracker.update(loss.item(), numSample)

        generated_codes += list(logits)
        label_train += list(labels)
        name_train += list(data['patchName'])
        

    train_writer.add_scalar("loss", lossTracker.avg, epoch)

    print('Train loss: {:.6f}'.format(lossTracker.avg))
    
    #s = torch.sum(model.FC.weight.data)
    #print('FC Sum Weight data: ',s)
    
    
    
    return (generated_codes,label_train,name_train)
</code></pre>
<p>And the model:</p>
<pre><code class="lang-auto">class ResNet50PairWise(nn.Module):
    def __init__(self, bits = 16):
        super().__init__()

        resnet = models.resnet50(pretrained=False)

        self.conv1 = nn.Conv2d(12, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
        self.encoder = nn.Sequential(
            self.conv1,
            resnet.bn1,
            resnet.relu,
            resnet.maxpool,
            resnet.layer1,
            resnet.layer2,
            resnet.layer3,
            resnet.layer4,
            resnet.avgpool
        )
        self.FC = nn.Linear(2048, bits)


        self.apply(weights_init_kaiming)
        self.apply(fc_init_weights)

    def forward(self, x):
        x = self.encoder(x)
        x = x.view(x.size(0), -1)

        logits = self.FC(x)
        sign = torch.sign(logits)
        binary_out = torch.relu(sign)

        return binary_out   
</code></pre>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>I’m not familiar with the mentioned loss function, but if you already checked the gradients, it would at least mean that the computation graph is not detached accidentally.</p>
<p>You could check, if the relu activation is needed at the model output, as this seems to be unusual (of course it might still fit your use case).</p>

username: Mr.Bank
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your message.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a86fb49291922c8c36e49a03a74868a025d08bd.png" alt="image" data-base62-sha1="fcnO8beeDo3uu4GgD3Eyn2yuK1n" width="457" height="97"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/515151b2192c3d8e8654e4368ab647a46e29e229.png" alt="image" data-base62-sha1="bBmTPCtfwNygYG8yTsWGzohogbL" width="288" height="65"></p>
<p>These are details about that loss function.<br>
c: number of common labels between binary pairs<br>
l: hash length</p>
<p>I use ReLu in the model because I want to produce only 0 and 1 from the network. That’s a hashing model so it should produce only binary results. That’s why sign and Relu have been used. Is it not an acceptable way?</p>
<p>Max function in loss function can not be differentiated in some cases as mentioned in</p><aside class="quote" data-post="1" data-topic="67501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/f1d935/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/use-max-operation-in-loss-function/67501">Use max operation in loss function</a> 
  </div>
  <blockquote>
    hi, im very confused. 
from all i know, mathematicly, max operation is not differential 
i read here a lot of contrasting answers - 
some people are saying that it is not differential, and therefor cant be used in a loss function 
and some are saying that it is differential “enough” for backpropagation 
does someone have an answer and a good explanation about that issue?
  </blockquote>
</aside>
<p>
That’s why I changed the max function with ReLu in loss function but it still causes the same problem.</p>
<p>I added these two lines to existed approach to see weight updates</p>
<pre><code class="lang-auto">        loss.backward()
        optimizer.step()
        
        s = torch.sum(model.FC.weight.data)
        print(s)
</code></pre>
<p>And it returns:<br>
Epoch 0:<br>
tensor(8.6537, device=‘cuda:0’)<br>
tensor(8.6536, device=‘cuda:0’)<br>
tensor(8.6535, device=‘cuda:0’)</p>
<p>Epoch 1:<br>
Tensor(8.6534, device=‘cuda:0’)<br>
tensor(8.6532, device=‘cuda:0’)<br>
tensor(8.6531, device=‘cuda:0’)</p>
<p>Epoch 2:<br>
tensor(8.6529, device=‘cuda:0’)<br>
tensor(8.6527, device=‘cuda:0’)<br>
tensor(8.6525, device=‘cuda:0’)</p>
<p>Does it mean loss function update weights correctly? But loss value is still not decreasing.</p>

username: ptrblck
cooked: <p>The <code>sign</code> method would return a zero gradient wouldn’t it?<br>
The only change in the output of <code>sign</code> would be at <code>x=0</code>, so that you might be killing the gradient with this method.</p>
<p>Are you sure the paper uses a <code>sign</code> method for its implementation?</p>
<p>Note that I’m just pointing to “unusual” model design choices, but it might be perfectly valid for your use case. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Mr.Bank
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your pointing out. Sign method is a zero gradient so it breaks the gradients.<br>
How can I produce only 1s and 0s from the network without that approach?</p>

username: ptrblck
cooked: <p><code>torch.sign</code> isn’t technically breaking the computation graph, but will return a zero gradient.<br>
I think you could use a <code>sigmoid</code> as a smooth approximation for values in <code>[0, 1]</code>.<br>
<code>tanh</code> would approximate the <code>sign</code> method or alternatively you could also use<br>
<code>x / (torch.sqrt(x*x + delta))</code>.</p>

username: Mr.Bank
cooked: <p>They will produce value in the range not only 0s and 1s. Is there any way to create only 0s and 1s?<br>
In the last function, you mentioned, how delta is calculated?</p>

username: ptrblck
cooked: <p>You could chose <code>delta</code> manually. The smaller it is, the more you would approximate the step function.</p>
<aside class="quote no-group" data-username="Mr.Bank" data-post="7" data-topic="87514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mr.bank/48/26210_2.png" class="avatar"> Mr.Bank:</div>
<blockquote>
<p>Is there any way to create only 0s and 1s?</p>
</blockquote>
</aside>
<p>Yes, with a threshold (step function), but then you would get zero gradients, as the derivative of the step function would be the delta function, which has only a valid value at <code>x=0</code> (if the step is there and not shifted).</p>