Equivalent tensorflow loss in pytorch

Gul_Zain: <p>tensorflow have the loss function “tf.nn.sparse_softmax_cross_entropy_with_logits”. I saw an implementation of log function like this</p>
<pre><code class="lang-auto"> def log_loss(logits,labels,name)
    SE_loss = tf.nn.sparse_softmax_cross_entropy_with_logits
    return tf.reduce_sum(SE_loss(logits=logits, labels=label), [1, 2], name=name)
</code></pre>
<p>What would be the equivalent loss in pytorch and how to use this loss to have exact same answer as of the log_loss function provided given same logits and labels tensors</p>
<p>Here is how i am testing it but the pytorch cross entropy loss is not working</p>
<pre><code class="lang-auto">input = torch.randn((20, 30, 30,2), requires_grad=True)
target = torch.empty((20,30,30), dtype=torch.long).random_(1)
output = loss(input, target)
logits= input.detach().numpy()
labels= target.detach().numpy()

graph = tf.Graph()
with tf.Session(graph=graph) as sess:
    output_tf= tf.nn.sparse_softmax_cross_entropy_with_logits(labels=tf.convert_to_tensor(labels, dtype=tf.int32), logits=tf.convert_to_tensor(logits, dtype=tf.float32))
    output_tf= tf.reduce_sum(output_tf, [1, 2], name="tfloss")

    print("*"*50)
    print("tensorflow: ",sess.run(output_tf))
    print(output)
    print("*"*50)#
</code></pre>
<p>I get this error</p>
<blockquote>
<p>RuntimeError: Assertion `input0 == target0 &amp;&amp; input2 == target1 &amp;&amp; input3 == target2’ failed. size mismatch (got input: 20x30x30x2, target: 20x30x30) at /pytorch/aten/src/THNN/generic/SpatialClassNLLCriterion.c:61</p>
</blockquote>
<p>When i run simply tensorflow part, i get following type of results.</p>
<hr>
<p>tensorflow:  [782.8956  814.34094 819.82416 814.04565 804.5945  817.7538  816.1295<br>
776.78687 785.20825 801.3092  802.24805 802.3621  846.7362  792.4359<br>
768.00555 820.16675 782.90857 846.25085 793.9464  799.1505 ]</p>
<hr>
<p>Length of the output is 20</p>
<p>I want to manipulate my input and output for pytorch cross entropy such that results of tensorflow code and pytorch cross entropy code come out as same. Looking forward to hearing from you. Thanks in advance for putting thought into it.</p>

ptrblck: <p>The class dimension should be in dim1 in your model output, so this code should work:</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss()
input = torch.randn((20, 2, 30, 30), requires_grad=True)
target = torch.empty((20,30,30), dtype=torch.long).random_(2)
output = criterion(input, target)
</code></pre>
<p>I’m not sure, if the TF code returns an unreduced loss, but if so you could set <code>reduction='none'</code> in your criterion instantiation.</p>

Gul_Zain: <p>it resolves the issue but the answer is not same as tensorflow. even the dimensions are not same as of tensorflow code. Not sure how cross entropy is working in terms of dimensions in pytorch. Code is given below.</p>
<pre><code class="lang-auto">import torch.utils.data as Data
import torchvision
from torch.autograd import Variable
from torchvision import transforms
import config as cfg
import os
from torch.utils.data import Dataset, DataLoader
import scipy.io as sio
from PIL import Image
import h5py
import numpy as np
import math
import cv2
from torchvision import transforms
from PIL import Image
import h5py
from tensorboardX import SummaryWriter
import tensorflow as tf

#
loss = nn.CrossEntropyLoss(reduction='none')

input = torch.randn((20, 30, 30,2), requires_grad=True)
target = torch.empty((20,30,30), dtype=torch.long).random_(1)

logits= input.detach().numpy()
labels= target.detach().numpy()
input= input.permute(0,3,1,2)

output = loss(input, target)

def log_loss(logits,labels,name):
    SE_loss = tf.nn.sparse_softmax_cross_entropy_with_logits
    return tf.reduce_sum(SE_loss(logits=logits, labels=labels), [1, 2], name=name)
import sys
graph = tf.Graph()
with tf.Session(graph=graph) as sess:

    output_tf= log_loss(tf.convert_to_tensor(logits, dtype=tf.float32),tf.convert_to_tensor(labels, dtype=tf.int32), name="loss")
    # output_tf=tf.reduce_mean(output_tf)#uncomment this to get reduced answer for tensorflow
    print("*"*50)
    print("tensorflow: ",sess.run(tf.shape(output_tf)))
    print("Pytorch: ",output.shape)
    print("*"*50)#

</code></pre>
<p>Output of the code is given below which shows the dimensions or two cross entropy function outputs.</p>
<pre><code class="lang-auto">**************************************************
tensorflow:  [20]
Pytorch:  torch.Size([20, 30, 30])
**************************************************
</code></pre>

ptrblck: <p>If seems TF reduced the loss, so you could either remove <code>reduction='none'</code> or set it to your desired reduction type. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>