How to write custom CrossEntropyLoss

username: alie
cooked: <p>I am learning Logistic Regression within Pytorch and to better understand I am defining a custom CrossEntropyLoss as below:</p>
<pre><code class="lang-auto">def softmax(x):
    exp_x = torch.exp(x)
    sum_x = torch.sum(exp_x, dim=1, keepdim=True)

    return exp_x/sum_x

def log_softmax(x):
    return torch.exp(x) - torch.sum(torch.exp(x), dim=1, keepdim=True)

def CrossEntropyLoss(outputs, targets):
    num_examples = targets.shape[0]
    batch_size = outputs.shape[0]
    outputs = log_softmax(outputs)
    outputs = outputs[range(batch_size), targets]

    return - torch.sum(outputs)/num_examples
</code></pre>
<p>I also make my own logistic regression (to predict FashionMNIST) as below:</p>
<pre><code class="lang-auto">input_dim = 784 # 28x28 FashionMNIST data
output_dim = 10

w_init = np.random.normal(scale=0.05, size=(input_dim,output_dim))
w_init = torch.tensor(w_init, requires_grad=True).float()
b = torch.zeros(output_dim)

def my_model(x):
    bs = x.shape[0]
    return x.reshape(bs, input_dim) @ w_init + b
</code></pre>
<p>To validate my custom crossentropyloss, I compared it with nn.CrossEntropyLoss from Pytorch by applying it on FashionMNIST data as below:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()

for X, y in trn_fashion_dl:
    outputs = my_model(X)
    my_outputs = softmax(outputs)

    my_ce = CrossEntropyLoss(my_outputs, y)
    pytorch_ce = criterion(outputs, y)

    print (f'my custom cross entropy: {my_ce.item()}\npytorch cross entroopy: {pytorch_ce.item()}')
    break 
</code></pre>
<p>My question is toward the results my_ce (my cross entropy) vs pytorch_ce (pytorch cross entropy) where they are different:</p>
<pre><code class="lang-auto">my custom cross entropy: 9.956839561462402
pytorch cross entroopy: 2.378990888595581
</code></pre>

username: mailcorahul
cooked: <p><a class="mention" href="/u/alie">@alie</a> There are two mistakes here.</p>
<ol>
<li>You apply softmax twice - once before calling your custom loss function and inside it as well.</li>
<li>You are not applying log to softmax output. Inside <code>log_softmax()</code>:<br>
<code>return torch.log(torch.exp(x) / torch.sum(torch.exp(x), dim=1, keepdim=True))</code>
</li>
</ol>

username: alie
cooked: <p><a class="mention" href="/u/mailcorahul">@mailcorahul</a> Thanks; after changing the log_softmax() function with yours, the two cross entropy beam closer but still they are not exactly the same. Is this expected or there is mistake somewhere else?</p>
<blockquote>
<p>my custom cross entropy: 2.319404125213623<br>
pytorch cross entroopy: 2.6645867824554443</p>
</blockquote>

username: mailcorahul
cooked: <p>Those two were the mistakes in the code. Can you post your full code again here?</p>

username: alie
cooked: <p>Thanks, sure here it is:</p>
<blockquote>
<p>import numpy as np<br>
import torch<br>
import torchvision<br>
from torchvision import transforms, datasets<br>
from torch.utils.data import DataLoader<br>
import torch.nn as nn</p>
<p>def softmax(x):<br>
exp_x = torch.exp(x)<br>
sum_x = torch.sum(exp_x, dim=1, keepdim=True)</p>
<pre><code>return exp_x/sum_x
</code></pre>
<p>def log_softmax(x):<br>
return x - torch.logsumexp(x,dim=1, keepdim=True)</p>
<p>def CrossEntropyLoss(outputs, targets):<br>
num_examples = targets.shape[0]<br>
batch_size = outputs.shape[0]<br>
outputs = log_softmax(outputs)<br>
outputs = outputs[range(batch_size), targets]</p>
<pre><code>return - torch.sum(outputs)/num_examples
</code></pre>
<p>def my_model(x):<br>
bs = x.shape[0]<br>
return x.reshape(bs, input_dim) @ w_init + b</p>
<p><span class="hashtag-raw">#FashionMNIST</span> Datasets for training/test<br>
trans = transforms.Compose([transforms.ToTensor()])<br>
trn_ds = datasets.FashionMNIST(‘./training’, download=True, transform=trans)<br>
test_ds = datasets.FashionMNIST(‘./test’, train=False, download=True, transform=trans)</p>
<p><span class="hashtag-raw">#Dataloader</span> for training/test<br>
trn_dl = DataLoader(trn_ds, batch_size=64, shuffle=True)<br>
test_dl = DataLoader(test_ds, batch_size=64)</p>
<p><span class="hashtag-raw">#paramaters</span> initialization<br>
input_dim = 784 # 28x28 FashionMNIST data<br>
output_dim = 10<br>
w_init = np.random.normal(scale=0.05, size=(input_dim,output_dim))<br>
w_init = torch.tensor(w_init, requires_grad=True).float()<br>
b = torch.zeros(output_dim)</p>
<p><span class="hashtag-raw">#pytorch</span> CrossEntropyLoss<br>
criterion = nn.CrossEntropyLoss()</p>
<p>for X, y in trn_dl:<br>
outputs = my_model(X)<br>
my_outputs = softmax(outputs)</p>
<pre><code>my_ce = CrossEntropyLoss(my_outputs, y)
pytorch_ce = criterion(outputs, y)

print (f'my custom cross entropy: {my_ce.item()}\npytorch cross entroopy: {pytorch_ce.item()}')

break
</code></pre>
</blockquote>

username: mailcorahul
cooked: <p><a class="mention" href="/u/alie">@alie</a> I am finding it difficult to understand the code because of its formatting, but I can see you’re applying softmax twice. You can either remove the line <code> my_outputs = softmax(outputs)</code> or replace the 3rd line in <code>CrossEntropyLoss()</code> to <code>outputs = torch.log(outputs)</code>.</p>

username: alie
cooked: <p>Thanks again, removing the softmax solves the problems; no both ce returns the exact same values</p>
<blockquote>
<p>my_outputs = softmax(outputs)</p>
</blockquote>