Batthacaryya loss

username: lelouedec
cooked: <p>Hello,</p>
<p>I would like to use the <a href="https://en.wikipedia.org/wiki/Bhattacharyya_distance" rel="nofollow noopener">Bhattacharyya</a> distance between two saliency maps as a loss function for my network. But if I want to use the autograd engine to back-propagate my loss over the network I need to keep my output as a Variable which doesn’t seems to be possible considering I need to do a element wise multiplication.</p>
<p>Any idea ?</p>
<p>Yours Justin</p>

username: jpeg729
cooked: <p>What about using <a href="http://pytorch.org/docs/0.3.0/torch.html#torch.addcmul"><code>torch.addcmul</code></a>?</p>

username: lelouedec
cooked: <p>Fair point but there is a square root <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>

username: jpeg729
cooked: <p><a href="http://pytorch.org/docs/0.3.0/torch.html#torch.sqrt"><code>torch.sqrt</code></a></p>

username: lelouedec
cooked: <p>I used the following:</p>
<blockquote>
<p>def bhatta_loss(output,target):<br>
out = -torch.log(torch.sum(torch.sqrt(torch.abs(torch.mul(output, target)))))<br>
return  out</p>
</blockquote>
<p>backward works but I get nan after the first iteration <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"><br>
Any ideas ?</p>

username: jpeg729
cooked: <p>The only way loss could be nan, would be if <code>torch.mul(output, target)</code> was comprised entirely of zeros. Maybe your model is rapidly learning to produce zero output.</p>

username: lelouedec
cooked: <p>I meant the outpput of  the  network becomes a nan on the next forward after having backpropagate.</p>

username: jpeg729
cooked: <p>Weird.<br>
Without seeing any code I have no idea what might be happening.</p>

username: lelouedec
cooked: <p>oops sorry:<br>
my network:</p>
<blockquote>
<p>class VGGNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
“”“Select conv1_1 ~ conv5_1 activation maps.”“”<br>
super(VGGNet, self).<strong>init</strong>()<br>
self.select = [15,22,29]<br>
self.features = torch.nn.Sequential(<br>
# conv1<br>
torch.nn.Conv2d(in_channels=3,out_channels=64, kernel_size=3,padding=35),<br>
torch.nn.ReLU(),<br>
torch.nn.Conv2d(64, 64, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.MaxPool2d(2, stride=2),<br>
# conv2<br>
torch.nn.Conv2d(64, 128, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.Conv2d(128, 128, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.MaxPool2d(2, stride=2),<br>
# conv3<br>
torch.nn.Conv2d(128, 256, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.Conv2d(256, 256, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.Conv2d(256, 256, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.MaxPool2d(2, stride=2),<br>
# conv4<br>
torch.nn.Conv2d(256, 512, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.Conv2d(512, 512, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.Conv2d(512, 512, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.MaxPool2d(2, stride=2),<br>
# conv5<br>
torch.nn.Conv2d(512, 512, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.Conv2d(512, 512, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.Conv2d(512, 512, 3, padding=1),<br>
torch.nn.ReLU(),<br>
torch.nn.MaxPool2d(2, stride=2)<br>
)<br>
self.deconv1 = torch.nn.Sequential(<br>
torch.nn.ConvTranspose2d(256, 128, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(128, 64, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(64, 1, 3, padding=0,stride=1),<br>
torch.nn.ReLU(),<br>
)<br>
self.deconv2 = torch.nn.Sequential(<br>
torch.nn.ConvTranspose2d(512, 256, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(256, 128, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(128, 64, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(64, 1, 3, padding=0,stride=1),<br>
torch.nn.ReLU(),<br>
)<br>
self.deconv3 = torch.nn.Sequential(<br>
torch.nn.ConvTranspose2d(512, 512, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(512, 256, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(256, 128, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(128, 64, 4, stride=2),<br>
torch.nn.ReLU(),<br>
torch.nn.ConvTranspose2d(64, 1, 3, padding=0,stride=1),<br>
torch.nn.ReLU(),<br>
)<br>
self.final_attention_pred = torch.nn.Sequential(<br>
torch.nn.ConvTranspose2d(9, 1, 3, stride=1,padding=1)<br>
)<br>
self._initialize_weights()</p>
<pre><code>def _initialize_weights(self):
    # initializing weights using ImageNet-trained model from PyTorch
    for i, layer in enumerate(models.vgg16(pretrained=True).features):
        if isinstance(layer, torch.nn.Conv2d):
            self.features[i].weight.data = layer.weight.data
            self.features[i].bias.data = layer.bias.data
    for m in self.deconv1:
        if isinstance(m, torch.nn.ConvTranspose2d):
            m.weight.data.normal_(0.0, 0.01)
            m.bias.data.zero_()  
            
    for m in self.deconv1:
        if isinstance(m, torch.nn.ConvTranspose2d):
            m.weight.data.normal_(0.0, 0.01)
            m.bias.data.zero_()  
            
            
    for m in self.deconv2:
        if isinstance(m, torch.nn.ConvTranspose2d):
            m.weight.data.normal_(0.0, 0.01)
            m.bias.data.zero_()  
            
    for m in self.deconv3:
        if isinstance(m, torch.nn.ConvTranspose2d):
            m.weight.data.normal_(0.0, 0.01)
            m.bias.data.zero_()  
            
    for m in self.final_attention_pred:
        if isinstance(m, torch.nn.ConvTranspose2d):
            m.weight.data.normal_(0.0, 0.01)
            m.bias.data.zero_()  

def forward(self, x):
    ##return list of feature map at different size
    features = []
    for i, layer in enumerate(self.features):
        layer.register_backward_hook(printgradnorm)
        if(i in self.select ):
            x = layer(x)
            features.append(x)
        else:
            x = layer(x)
    for i in self.deconv1:
        i.register_backward_hook(printgradnorm)
    
    for i in self.deconv2:
        i.register_backward_hook(printgradnorm)
        
    for i in self.deconv3:
        i.register_backward_hook(printgradnorm)
        
    self.final_attention_pred[0].register_backward_hook(printgradnorm)
        
    saliency = [] 
    m = nn.Sigmoid()
    m1 = nn.Sigmoid()
    m2 = nn.Sigmoid()
    m3 = nn.Sigmoid()
    
   
    m.register_backward_hook(printgradnorm)
    attentionmap1 = self.deconv1(features[0])[:, :, 36:260, 36:260]
    attentionmap1 = [attentionmap1,attentionmap1,attentionmap1]
    attentionmap1 = torch.cat(attentionmap1,1)
    
    attentionmap2 = self.deconv2(features[1])[:, :, 42:266, 42:266]
    attentionmap2 = [attentionmap2,attentionmap2,attentionmap2]
    attentionmap2 = torch.cat(attentionmap2,1)

    
    attentionmap3 = self.deconv3(features[2])[:, :, 54:278, 54:278]
    attentionmap3 = [attentionmap3,attentionmap3,attentionmap3]
    attentionmap3 = torch.cat(attentionmap3,1)


    
    saliency.append(m(attentionmap1))
    
    saliency.append(m1(attentionmap2))

    saliency.append(m2(attentionmap3))

    output_data = torch.cat(saliency,1)
    output  = m3(self.final_attention_pred(output_data))
    return output
</code></pre>
</blockquote>
<p>and the function of loss is above</p>

username: jpeg729
cooked: <p>I have to say I am not that experienced with image processing using neural networks. You might be better off starting a new thread to ask why the network might output NaN.</p>
<p>An alternative would be to stick a load of print statements in the forward method in order to try and figure out at what point in the calculation the NaNs start appearing.</p>

username: natank1
cooked: <p>I encounter the same issue with the same metric. How did you resolve it?</p>

username: nkamath
cooked: <p>To add to what <a class="mention" href="/u/jpeg729">@jpeg729</a> pointed out,</p>
<blockquote>
<p>The only way loss could be nan, would be if <code>torch.mul(output, target)</code> was comprised entirely of zeros.</p>
</blockquote>
<p>or if <code>output</code> or <code>target</code> have a NaN,<br>
or if they have valid values but disjoint supports (one element in element-wise mul is always 0),</p>
<p>torch.log() would return NaN. A workaround for the latter case is to add a small epsilon (say, 1e-6) to the tensor inside the log.</p>

username: dherrera1911
cooked: <p>The square root can also lead to numerical instabilities when doing gradient descent, so maybe adding the epsilon inside the sqrt would be best?</p>