Vanishing gradients

David_Jitca: <p>Hello. I have a Unet network for segmentation. Using dice loss  the net is learning. But when I use Crossentropy, after one epoch the loss is constant. I ve searched on internet and i ve found a function to check gradients, and after 3 epochs they become 0. Can someone help me with some ideas?<br>
I already use Relu, and init the conv2d with xavier.</p>

JuanFMontesinos: <p>You can try to use leaky relu. I used it with unet and never had problems.</p>

David_Jitca: <p>Ok, but do you have an explanation, why with dice loss is no problem and with crossentropy it is?</p>

JuanFMontesinos: <p>Hmm it’s difficult to say. In theory cross entropy should has better gradient properties. You can find a post comparing both here <a href="https://becominghuman.ai/investigating-focal-and-dice-loss-for-the-kaggle-2018-data-science-bowl-65fb9af4f36c" rel="nofollow noopener">https://becominghuman.ai/investigating-focal-and-dice-loss-for-the-kaggle-2018-data-science-bowl-65fb9af4f36c</a>.</p>
<p>Anyway could you paste couple of loss plots?. If you report small gradients at an early stage  I assume there exist a smooth plateau in you loss for a high loss value. If you had big gradients but no convergence you would have oscillating loss and probably it’s not the case. Have you tried different set of initializations/lr schemes?</p>

David_Jitca: <p>I have constant loss.<br>
For example for adam optimiser with:<br>
lr = 0.01 the loss is 25 in first batch and then constanst 0,06x  and gradients after 3 epochs<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/023a027ceeba7c4eab31c2f2f1b1bdb52a863682.png" alt="Average_gradients" data-base62-sha1="jHeTi48XVMGW10PUH0bELeqPVU" width="487" height="378"> . But 0 accuracy.</p>
<p>lr = 0.0001 the loss is 25 in first batch and then constant 0,1x and gradients after 3 epochs<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5db05125725d73d68422efb4cc6196c614a1fe49.png" alt="average_2" data-base62-sha1="dmOa3f6Qm51aQPC3DQeZ9JjMOMh" width="487" height="378"></p>
<p>lr = 0.00001 the loss is 1 in first batch and then after 6 epochs constant <img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e1d6e66932c0b42f31f703144204a84de2467fde.png" alt="image" data-base62-sha1="wdRQPlUYVhCKaogM00sd4fAx77U" width="655" height="260"><br>
and gradients i dont know if thery are too small <img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f8c04a746047ec719aa2f339f33ec867a4795f28.png" alt="image" data-base62-sha1="zuyoq7yV9Ht1nVbeGi2G1eLqkdi" width="473" height="330"></p>

Nikronic: <p>Can you please share me what function you used to check gradients?<br>
It is a little hard on my environment to work with tensorboard.</p>

David_Jitca: <pre><code class="lang-auto">def plot_grad_flow(named_parameters):
    '''Plots the gradients flowing through different layers in the net during training.
    Can be used for checking for possible gradient vanishing / exploding problems.
    
    Usage: Plug this function in Trainer class after loss.backwards() as 
    "plot_grad_flow(self.model.named_parameters())" to visualize the gradient flow'''
    ave_grads = []
    max_grads= []
    layers = []
    for n, p in named_parameters:
        if(p.requires_grad) and ("bias" not in n):
            layers.append(n)
            ave_grads.append(p.grad.abs().mean())
            max_grads.append(p.grad.abs().max())
    plt.bar(np.arange(len(max_grads)), max_grads, alpha=0.1, lw=1, color="c")
    plt.bar(np.arange(len(max_grads)), ave_grads, alpha=0.1, lw=1, color="b")
    plt.hlines(0, 0, len(ave_grads)+1, lw=2, color="k" )
    plt.xticks(range(0,len(ave_grads), 1), layers, rotation="vertical")
    plt.xlim(left=0, right=len(ave_grads))
    plt.ylim(bottom = -0.001, top=0.02) # zoom in on the lower gradient regions
    plt.xlabel("Layers")
    plt.ylabel("average gradient")
I found this on forum
</code></pre>

JuanFMontesinos: <p>I does not look like you are getting gradient vanishing. In fact it is difficult with unet due to skip connections. It also looks like that even decreasing lr you are arriving to same local minima, thus, a bigger LR at early stage does not help not to fall in there. Could be a matter of trying different initialization/optimizers in order to see if the lead to another place?</p>
<p>Anyway, do you have a healthy dataset with balanced amount of classes?</p>

David_Jitca: <p>It’s a segmantation task. I have 2.5k images for traning. A gts containg 90% background and 10% tumor, or less I think</p>

JuanFMontesinos: <p>So you are using BCE with logits rather than crossentropy right? Have you tried to penalize the weight of background? It sounds like dice loss works as its kind of IoU but with crossentroypy model starts predicting all as background and it would be accurate loss-wise.</p>

David_Jitca: <p>I only use dice loss. Is there a loss that penalize weigth, bootstrapping loss no?I havent use it yet.</p>

JuanFMontesinos: <p>You started the post saying</p>
<aside class="quote no-group" data-username="David_Jitca" data-post="1" data-topic="46824">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_jitca/48/11746_2.png" class="avatar"> David_Jitca:</div>
<blockquote>
<p>But when I use Crossentropy, after one epoch the loss is constant</p>
</blockquote>
</aside>
<p>As far as I understand, your problem was crossentropy does not train but dice does. So one possible reason is that you should use BCE rather than crossentropy (as it is a binary classification, background vs foreground) and to penalize the weight with wich background contributes to the loss as it is 90% cases (thus, net will be biased to predict background).</p>

David_Jitca: <p>Hello. I dont know if is a problem with the arhitecture.<br>
But I have a new problem with a loss function, I found an implementation for bootstrap loss`class SoftBootstrappingLoss(nn.Module):<br>
“”"<br>
Loss(t, p) = - (beta * t + (1 - beta) * p) * log§<br>
“”"<br>
def <strong>init</strong>(self, beta=0.9, reduce=True):<br>
super(SoftBootstrappingLoss, self).<strong>init</strong>()<br>
self.beta = beta<br>
self.reduce = reduce</p>
<pre><code>def forward(self, input, target):
    target = target.long()
    target = target.view(-1, size_image, size_image)
    # cross_entropy = - t * log(p)
    beta_xentropy = self.beta * F.cross_entropy(input, target, reduce=False)

    # second term = - (1 - beta) * p * log(p)
    bootstrap = - (1.0 - self.beta) * torch.sum(F.softmax(input, dim=1) * F.log_softmax(input, dim=1), dim=1)

    if self.reduce:
        return torch.mean(beta_xentropy + bootstrap)
    return beta_xentropy + bootstrap
</code></pre>
<p>criterion = SoftBootstrappingLoss()`</p>
<p>But i obtain negative values for loss.</p>

henrique: <p>Are your weights converging to 0 as well when using BCE?<br>
If you’re using batch norm, what are the weights there?<br>
I’d suspect your batch norm sigma would quickly move towards 0 to satisfy the unbalanced dataset. I.e. if you always predict non-tumor you already get a quite good loss (an obvious local minima).</p>
<p>Most people use focal loss nowadays instead of cross entropy. If you have a look at the original paper from retinanet (which introduces focal loss) they also use a scalar to normalise the known unbalance. E.g. you basically want to increase your tumor loss 9-fold to match the non-tumor loss. Does it make sense?</p>