Error in the information entropy implementation!

username: External_happy
cooked: <p>Hello, I’m currently working on fine-tuning a pre-trained RN50 model that was initially trained on the ImageNet dataset. My goal is to optimize its performance on the Caltech dataset by reducing the information entropy.</p>
<pre><code class="lang-auto">softmax_out = F.softmax(logits, dim=1)
entropy = -softmax_out * torch.log(softmax_out + 1e-5)
loss = torch.mean(torch.sum(entropy, dim=1))
</code></pre>
<p>However, I encountered this issue even after training for all 50 epochs. I attempted to adjust the learning rate, but no improvements were observed.</p>
<p>`epoch [1/50] batch [1/2] loss 1.8028 acc 21.8750<br>
epoch [1/200] batch [2/2] loss 1.0584  acc 6.2500<br>
epoch [2/200] batch [1/2] loss 1.8191  acc 3.1250<br>
epoch [2/200] batch [2/2] loss 0.8207  acc 0.0000<br>
epoch [3/200] batch [1/2]  loss 0.0007  acc 3.1250<br>
epoch [3/200] batch [2/2]  oss 0.1916  acc 3.1250<br>
epoch [4/200] batch [1/2]  loss -0.0000 acc 3.1250<br>
epoch [4/200] batch [2/2]  loss -0.0000  acc 3.1250<br>
epoch [5/200] batch [1/2]   loss -0.0000  acc 3.1250<br>
epoch [5/200] batch [2/2]  loss 0.2917  acc 3.1250<br>
epoch [6/200] batch [1/2] loss 0.3784 acc 3.1250</p>

username: KFrank
cooked: <p>Hi External!</p>
<aside class="quote no-group quote-modified" data-username="External_happy" data-post="1" data-topic="189445" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/ecae2f/48.png" class="avatar"> External_happy:</div>
<blockquote>
<p>My goal is to optimize its performance on the Caltech dataset by reducing the information entropy.<br>
…<br>
However, I encountered this issue even after training for all 50 epochs.</p>
</blockquote>
</aside>
<p>I can’t say that I’m surprised by the result.  Given that you’ve only shown us<br>
about six epochs, I probably wouldn’t be surprised by almost any result.  Even<br>
fifty epochs wouldn’t really be a lot.</p>
<p>You haven’t said much about your use case or how you’re training.</p>
<p>Try training with plain-vanilla <code>SGD</code> (with no momentum nor weight decay)<br>
with a very small learning rate and train, potentially, for a long time.  Unless<br>
your model is weird somehow or you’re freezing the final layers of the model<br>
while you fine tune, you should be able to train down to zero “information<br>
entropy.”</p>
<p>When your probabilities, <code>softmax_out</code>, saturate with one of them (along a<br>
given batch-element row) being very close to one and all the others being<br>
very close to zero, your <code>loss</code> will attain its minimum of zero.  However, let’s<br>
say that the length of a row (<code>softmax_out.size (1)</code>) is <code>n</code>.  Then there will<br>
be <code>n</code> different ways you can minimize the loss (for that row), corresponding<br>
to which of the <code>n</code> probabilities approaches one.</p>
<p>Fine tuning to the point that you converge to a specific one of these equal-loss<br>
minima is probably not what you want.  So either your proposed scheme doesn’t<br>
fit your use case or you only want to fine tune a little bit, mostly preserving the<br>
weights and predictions of the pre-trained model.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">softmax_out = F.softmax(logits, dim=1)
entropy = -softmax_out * torch.log(softmax_out + 1e-5)
</code></pre>
</blockquote>
</aside>
<p>As an aside, consider, for reasons of numerical stability, using <code>log_softmax()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">entropy = -torch.softmax (logits, dim = 1) * torch.log_softmax (logits, dim = 1)
</code></pre>
<p>(Note, I don’t think this is relevant to your current issue.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: External_happy
cooked: <p>Hi KFrank,<br>
Thanks for the response. After a few days struggling with the code, I found out that my model is facing catastrophic forgetting. That’s why there was no improvement.</p>