Backward of crossentropyloss

Martina_Ragulikova: <p>Hi, this is crossentropyloss</p>
<pre><code class="lang-auto">#convert class ID to one hot encoding
one_hot = torch.zeros_like(y_pred)
one_hot[range(y_target.shape[0]), y_target] = 1.0

ce = -one_hot*(y_pred - y_pred.exp().sum(1).log().unsqueeze(1))
result = ce.sum(dim=1).mean()
</code></pre>
<p>Can someone help me with making derivative of this with respect to y_pred in python code? I just can not figure it out… What is backward of this losses function?</p>
<p>Thank you for help.</p>

rinkujadhav2013: <p>PyTorch will automatically calculate the derivative for you when you do <code>.backward()</code>. But I guess you’re looking for something else?</p>

Martina_Ragulikova: <p>Hi, i know backward can do it. But iam loking for manually derived function.</p>

rinkujadhav2013: <p>I see. It’s probably better to write the equations here in mathematical terms, show us where you get stuck while applying chain rule and we can take it from there.<br>
If you’re having difficulty with CE derivative then you’ll easily find some blogs and stuff talking about it’s derivative like:</p><aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/370723/how-to-calculate-the-derivative-of-crossentropy-error-function">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/370723/how-to-calculate-the-derivative-of-crossentropy-error-function" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/14144/cyberplayerone" target="_blank" rel="noopener nofollow ugc">
    <img alt="CyberPlayerOne" src="https://www.gravatar.com/avatar/ff849189a159031ae4338fb1787cde74?s=256&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/370723/how-to-calculate-the-derivative-of-crossentropy-error-function" target="_blank" rel="noopener nofollow ugc">How to calculate the derivative of crossentropy error function?</a>
</h4>

<div class="tags">
  <strong>machine-learning, neural-networks, optimization, loss-functions</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/14144/cyberplayerone" target="_blank" rel="noopener nofollow ugc">
    CyberPlayerOne
  </a>
  on <a href="https://stats.stackexchange.com/questions/370723/how-to-calculate-the-derivative-of-crossentropy-error-function" target="_blank" rel="noopener nofollow ugc">09:10AM - 08 Oct 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
or</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://sefiks.com/2017/12/17/a-gentle-introduction-to-cross-entropy-loss-function/">
  <header class="source">
      <img src="https://i0.wp.com/sefiks.com/wp-content/uploads/2018/03/cropped-cropped-sefiks-icon-v2.png?fit=32%2C32&amp;ssl=1" class="site-icon" width="32" height="32">

      <a href="https://sefiks.com/2017/12/17/a-gentle-introduction-to-cross-entropy-loss-function/" target="_blank" rel="noopener nofollow ugc" title="11:46AM - 17 December 2017">Sefik Ilkin Serengil – 17 Dec 17</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://sefiks.com/2017/12/17/a-gentle-introduction-to-cross-entropy-loss-function/" target="_blank" rel="noopener nofollow ugc">A Gentle Introduction to Cross-Entropy Loss Function - Sefik Ilkin Serengil</a></h3>

  <p>Neural networks produce multiple outputs in multiclass classification problems. However, they do not have ability to produce exact outputs, they can only produce continuous results. We would apply some additional steps to transform continuos results...</p>

  <p>
    <span class="label1">Est. reading time: 3 minutes</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Martina_Ragulikova: <p>Hi thank you, lets think about MSE loss:<br>
This is forward:</p>
<pre><code class="lang-auto">((labels - prediction) ** 2).mean()
</code></pre>
<p>And this is backward:</p>
<pre><code class="lang-auto">N = labels.shape[0]
first_grad = -2*(labels - prediction) / N
</code></pre>
<p>First grad is then used to backpropagate next layer and get  gradients of weights and bias.</p>
<p>The question is how to do the same with cross entropy loss?<br>
This is forward of it:</p>
<pre><code class="lang-auto">
ce = -labels*(y_pred - y_pred.exp().sum(1).log().unsqueeze(1))
result = ce.sum(dim=1).mean()
</code></pre>
<p>How to do backward from this?</p>

rinkujadhav2013: <aside class="quote no-group quote-modified" data-username="Martina_Ragulikova" data-post="5" data-topic="169740">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/martina_ragulikova/48/30971_2.png" class="avatar"> Martina_Ragulikova:</div>
<blockquote>
<p><code>ce = -one_hot*(y_pred - y_pred.exp().sum(1).log().unsqueeze(1))</code></p>
</blockquote>
</aside>
<p>I’m not sure if your implementation of CE loss is correct. You need to multiply <code>y_pred</code> with the one-hot encoding before taking the sum. Check the definition (eq. 1) of CE here: <a href="https://en.wikipedia.org/wiki/Cross_entropy#Definition" class="inline-onebox" rel="noopener nofollow ugc">Cross-entropy - Wikipedia</a> or here <a href="https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html" class="inline-onebox" rel="noopener nofollow ugc">Loss Functions — ML Glossary documentation</a></p>
<p>You can also call PyTorch’s CE loss function and ensure that you’re getting the same result via your code and PyTorch’s implementation.</p>

rinkujadhav2013: <p>The question is still whether you’re struggling with the math or the implementation.<br>
Also note that CE loss is usually used in conjunction with a softmax layer because the derivative of softmax+CE is really nice and easy to implement.</p>

Martina_Ragulikova: <p>I am struggling with implementation, I can not write backward. this forward is going 1:1 with pytorch cross entropy. so forward is good.</p>
<p>Here is proof, results are same as torch crossentropy:</p>
<pre><code class="lang-auto">import torch
def CustomCrossEntropyB(y_pred, y_target):
    #convert class ID to one hot encoding
    one_hot = torch.zeros_like(y_pred)
    one_hot[range(y_target.shape[0]), y_target] = 1.0

    #compute cross entropy without softmax
    ce = -one_hot*(y_pred - y_pred.exp().sum(1).log().unsqueeze(1))

    #reduction mean
    result = ce.sum(dim=1).mean()
    return result


if __name__ == "__main__":
    batch_size      = 3
    classes_count   = 10
    target_class_id = torch.randint(0, classes_count, (batch_size, ))

    #initial logits value
    x_initial = torch.randn((batch_size, classes_count))

    #reference, computed using torch cross entropy loss
    xr = torch.nn.Parameter(x_initial.clone(), requires_grad=True)

    loss_func_r = torch.nn.CrossEntropyLoss()
    optimizer_r  = torch.optim.Adam([xr], lr=0.5)


    #custom, computed using cusotm cross entropy loss
    xc = torch.nn.Parameter(x_initial.clone(), requires_grad=True)

    loss_func_c = CustomCrossEntropyB
    optimizer_c = torch.optim.Adam([xc], lr=0.5)

    #find (train) logits
    for i in range(100):

        optimizer_r.zero_grad()
        loss_r = loss_func_r(xr, target_class_id)
        loss_r.backward()
        optimizer_r.step()

        optimizer_c.zero_grad()
        loss_c = loss_func_c(xc, target_class_id)
        loss_c.backward()
        optimizer_c.step()
        
        print(loss_r, loss_c)

    print("\n\n")
    print("final logis values : ")
    print(xr)
    print(xc)
</code></pre>

Martina_Ragulikova: <p>So, I build my own backward of linear layers and activation functions, and I was using only MSE with that, now I want to add Crossentropy and I am not able to make its backward.</p>

rinkujadhav2013: <p>The answer in simple terms is that the derivative would be <code>softmax(y_pred)/batch_size</code> when <code>y_pred ≠ y_target</code> and it will be <code>(softmax(y_pred) - 1)/batch_size</code> when <code>y_pred==y_target</code>.<br>
You can try it in PyTorch.<br>
The equation basically translates to:<br>
<code>(F.softmax(a, dim=1)/a.shape[0]) * (1- F.one_hot(y_t)) + (F.softmax(a, dim=1) - 1)*F.one_hot(y_t)/a.shape[0]</code><br>
where <code>a</code> is the input to the CE loss and <code>y_t</code> represents the target indices.<br>
Or more simply:<br>
<code>(F.softmax(a, dim=1) - F.one_hot(y_t))/a.shape[0]</code></p>

shivammehta007: <p>The backwards of cross entropy is as simple as  <code>logits - predictions</code> and (scale it for the reduction i.e mean, sum or weighted mean), where logits are the output of the softmax layer and predictions are the one hot encoded labels. So basically</p>
<pre><code class="lang-python">first_grad = (softmax(prediction) - labels) / N
</code></pre>

shivammehta007: <p>Also, I tried to add the derivation of why this is the answer: <a href="https://shivammehta25.github.io/posts/deriving-categorical-cross-entropy-and-softmax" class="inline-onebox" rel="noopener nofollow ugc">Deriving categorical cross entropy and softmax | Shivam Mehta</a> .</p>