An issue with custom loss

yash1994: <p>Hi,</p>
<p>I have got a task (video analysis) where I have to divide the last fully connected layer of <code>(1+2)D ResNet</code> into multiple chunks and perform classification loss calculation on each individual chunk using <code>CrossEntropyLoss</code>. With more context, I’ve reshaped my model’s last layer (FC) to generate the output of shape (64, 4, 21) where 64 is the batch size, 4 is the number of chunks and 21 is the number of classes to perform classification. So basically my model is computing 4 class values for each video segment. I am computing loss in the following manner:</p>
<pre><code class="lang-python">def calculate_loss(output, target, criterion):
    loss = 0
    for i in range(output.size(1)):
        loss += criterion(output[:, i, :], target[:,i])
    return loss # / output.size(1)
</code></pre>
<p>Where I am iterating over each chunk and calculate the classification loss. My model trains well for a few batches (150/2500) but then loss and accuracy both stagnate. And when I check the output of each chunk it produces homogeneous classification values (almost same class), regardless of the target having different classes in each chunk.</p>
<p>Questions:</p>
<ol>
<li>Is this kind of loss calculation is permissible?</li>
<li>How does the backdrop happen in this scenario?</li>
</ol>

KFrank: <p>Hello Yash!</p>
<aside class="quote no-group" data-username="yash1994" data-post="1" data-topic="77807" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yash1994/48/21611_2.png" class="avatar"> yash1994:</div>
<blockquote>
<p>I am computing loss in the following manner:</p>
<pre data-code-wrap="python"><code class="lang-python">def calculate_loss(output, target, criterion):
    loss = 0
    for i in range(output.size(1)):
        loss += criterion(output[:, i, :], target[:,i])
    return loss # / output.size(1)
</code></pre>
</blockquote>
</aside>
<p>As an aside, you can use the “K-dimensional case” feature of<br>
<code>CrossEntropyLoss</code> (if you’re using a recent enough version of<br>
pytorch) to eliminate your <code>for</code> loop.</p>
<pre><code class="lang-auto">output.size (1) * torch.nn.CrossEntropyLoss() (output.transpose (1, 2), target)
</code></pre>
<p>should yield the same result as your <code>calculate_loss</code> function.</p>
<aside class="quote no-group">
<blockquote>
<p>Questions:</p>
<ol>
<li>Is this kind of loss calculation is permissible?</li>
</ol>
</blockquote>
</aside>
<p>Yes, there is nothing in principle wrong with what you are doing.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>How does the backdrop happen in this scenario?</li>
</ol>
</blockquote>
</aside>
<p>The same as usual.  <code>output</code> is a differentiable function of your model<br>
parameters, the <code>loss</code> in your loop for each chunk is a differentiable<br>
function of <code>output</code>, and summing <code>loss</code> over the four chunks is<br>
differentiable, so the gradients backpropagate through your summed<br>
<code>loss</code> all the way back to your model parameters.</p>
<aside class="quote no-group">
<blockquote>
<p>My model trains well for a few batches (150/2500) but then loss and accuracy both stagnate. And when I check the output of each chunk it produces homogeneous classification values (almost same class), regardless of the target having different classes in each chunk.</p>
</blockquote>
</aside>
<p>Many things could cause this, but it’s not caused directly by your<br>
“chunked-loss” function.  There could be some bug in your model,<br>
or it could just be that the problem you’re working on is hard (or<br>
impossible), and the way your training data and model interact<br>
makes training difficult.</p>
<p>Best.</p>
<p>K. Frank</p>

yash1994: <p>Thanks for reply <a class="mention" href="/u/kfrank">@KFrank</a>. I looked at data more carefully and found some anomalies in data itself so that might be the reason for poor training. Thanks again for clearing my doubts.</p>