Loss functions for batches

Alex_Ge: <p>Hi,</p>
<p>Apologies if this seems like a noob question; I’ve read similar issues and their responses and looked at all the related examples.</p>
<p>I’m really confused about what the expected predicted and ideal arguments are for the loss functions. I’m building a CNN for image classification and there are 4 possible classes.<br>
Trying to use <code>nn.CrossEntropyLoss</code> I get errors:</p>
<pre><code class="lang-auto">RuntimeError: multi-target not supported at ClassNLLCriterion.cu:16
</code></pre>
<p>The batch size is 10, and the labels are 4. So the <code>outputs</code> tensor is <code>10 x 4</code>, and so is the <code>labels</code> tensor.</p>
<p>The code is (minified):</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss().cuda()
print("using CE loss")

optimizer = torch.optim.SGD(model.parameters(),
                            lr=learning_rate,
                            momentum=0.9,
                            dampening=0,
                            weight_decay=l2_reg,
                            nesterov=False)

model.train()
total_step = len(train_loader)

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        inputs = images.cuda().half()
        labels = labels.cuda().long()
        optimizer.zero_grad()
        outputs = model(inputs)
        print(labels.size())
        print(outputs.size())
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
</code></pre>
<p>I understand that the <code>criterion</code> expects to see a 1D tensor, and so I’ve tried reducing it:</p>
<pre><code class="lang-auto">print(labels.size())
ideals = labels.view(-1) 
print(ideals.size())
loss = criterion(outputs, ideals)
</code></pre>
<p>So now the sizes are:</p>
<p>labels: (10, 4)<br>
ideals: (40,)</p>
<p>And now I get the error of:</p>
<pre><code class="lang-auto">ValueError: Expected input batch_size (10) to match target batch_size (40)
</code></pre>
<p>Which makes sense. So what I don’t get is:</p>
<ul>
<li>do I need to iterate the ideal and actual output for each input in the batch?</li>
<li>or do I need to reduce both tensors to the same size and dimensions?</li>
</ul>
<p>Most of the examples seems to be using directly</p>
<pre><code class="lang-auto">loss = criterion(outputs, labels)
</code></pre>
<p>And the only exception I’ve seen is at the official documentation (<a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#training-on-gpu" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#training-on-gpu</a>) where the labels are:</p>
<pre><code class="lang-auto">_, predicted = torch.max(outputs, 1)
</code></pre>
<p>But sadly I don’t see an explanation?</p>
<p>Many thanks!<br>
Alex</p>

ptrblck: <p>The loss functions for classification, e.g. <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code>, require your target to store the class indices instead of a one-hot encoded <code>tensor</code>.<br>
So if your target looks like:</p>
<pre><code class="lang-python">labels = torch.tensor([[0, 1, 0],
                       [1, 0, 0],
                       [0, 0, 1]])
</code></pre>
<p>you would have to get the corresponding indices by:</p>
<pre><code class="lang-python">labels = labels.argmax(1)
print(labels)
&gt; tensor([1, 0, 2])
</code></pre>
<p>Now you can use this target <code>tensor</code> for your criterion.</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss()
x = torch.randn(3, 3, requires_grad=True)

loss = criterion(x, labels)
loss.backward()
</code></pre>
<p>Your code example calculates just the most probably class and stores its index in <code>predicted</code>.<br>
This is done to calculate the accuracy or just get the most likely class for the current sample.</p>

Alex_Ge: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thank you very much, this worked and makes perfect sense.<br>
I’m just wondering, did I miss the documentation/example, is this somewhere described in detail? Also, I am guessing that <code>nn.CrossEntropyLoss</code> uses <code>Softmax</code> internally?</p>
<p>Again, many Thanks!</p>

ptrblck: <p>The usage, shapes and types are described in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">docs</a>.<br>
It uses <code>nn.LogSoftmax</code> and <code>nn.NLLLoss</code> internally, so you should pass the logits into the criterion.</p>

JBoRu: <p>Excuse me, I have one experiment named “opinion and aspect co-extraction” which judges whether the word in a sentence is opinion and aspect or not, and labels them as follows:</p>
<blockquote>
<p>0：the background word<br>
1：the begin of an aspect word<br>
2:   the include of an aspect word<br>
3：the begin of an opinion word<br>
4：the include of an opinion word</p>
</blockquote>
<p>Therefore, there are five classes for each word in a sentence.And the example of input_x(after word to index) and output(after logsoftmax) and y(gold label) of one batch of sentences like that:</p>
<blockquote>
<p>input_x: batch_size x sentence_length<br>
output: batch_size x sentence_length x class_possibility<br>
y: batch_size x sentence_length</p>
</blockquote>
<p>After I input the input_x and train one epoch and get the output, i use the nn.NLLLoss to compute the loss between output and y,but it occured some error. The next I will use some code simulate the situation.</p>
<pre><code class="lang-auto"># the output of one batch of 10 sentences, every sentence has 78 words, and every word has a score after logsoftmax
p = torch.rand(10,78,5)
# the gold label of one batch of 10 sentences, every sentence has 78 words, and each word have one index of 0~4 which indicate its property, the detail meaning as above 
y = torch.ones(10,78).long()
# use the NLLLoss function
loss = nn.NLLLoss()
# get the loss value
r = loss(p,y)
</code></pre>
<p>and it report such error:</p>
<blockquote>
<p>ValueError: Expected target size (10, 5), got torch.Size([10, 78])</p>
</blockquote>
<p>so, i want to how can i cmpute the loss between the output and gold label of a batch?</p>

ptrblck: <p>The class logits should be in dim1 in your use case, so you could permute your output as:</p>
<pre><code class="lang-python">p = p.permute(0, 2, 1)
</code></pre>
<p>or alternatively make sure your model outputs the right shapes, but this of course depends on the architecture. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

JBoRu: <p>I’m so sorry that is such a simple question!<br>
But, I have read the pytorch docs, i can’t find the useful information to sovle my question, I wonder if the red circle in the picture below can solve my problem?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/127161eddb2dede6563e3362259440e24a4ab441.png" data-download-href="https://discuss.pytorch.org/uploads/default/127161eddb2dede6563e3362259440e24a4ab441" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/127161eddb2dede6563e3362259440e24a4ab441_2_690x173.png" alt="image" data-base62-sha1="2D9vXjuieebCJL57dm9l0xdqDpT" width="690" height="173" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/127161eddb2dede6563e3362259440e24a4ab441_2_690x173.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/127161eddb2dede6563e3362259440e24a4ab441_2_1035x259.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/127161eddb2dede6563e3362259440e24a4ab441_2_1380x346.png 2x" data-dominant-color="0E0C0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1503×378 41.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
However, i’m sorry that i can’t understand the meanings of “K”, what means “in the case of K-dimensional loss”?Thanks for you patient reply!</p>

ptrblck: <p><code>K</code> is a placeholder for the number of additional dimensions your output and target have.<br>
In a simple classification use case, <code>K</code> would be <code>0</code>, which means:</p>
<ul>
<li>
<code>output = [batch_size, nb_classes]</code>, <code>target = [batch_size]</code>
</li>
</ul>
<p>In the case of <code>K=1</code>, e.g. for a temporal signal, where each sample belongs to a specific class:</p>
<ul>
<li>
<code>output = [batch_size, nb_classes, seq_len]</code>, <code>target = [batch_size, seq_len]</code>
</li>
</ul>
<p>For a segmentation use case:</p>
<ul>
<li>
<code>output = [batch_size, nb_classes, height width]</code>, <code>target = [batch_size, height width]</code>
</li>
</ul>
<p>…<br>
As you can see, <code>K</code> simply indicates the dimensionality of your current use case and how the output and target should look like.</p>

JBoRu: <p>I’m so thanks for your simple and useful reply!<br>
Best wish for you！</p>