Catrogircal cross entropy with soft classes

username: John_Deterious
cooked: <p>I looked  @ <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html</a><br>
And I failed to find what I wanted. I basically have multiclass problem, and the targets for each instance is a vector of soft classes.<br>
To be concrete:<br>
nueral net output [0.1, 0.5, 0.4]<br>
correct label [0.2, 0.4, 0.4]<br>
It should be standard in library.</p>
<p>I created my simple Python function to do it, but can anyone please help me wrap it as a proper pytorch loss function?</p>
<pre><code class="lang-auto">&gt; def CXE(predicted, target):
&gt;     return -(target * t.log(predicted)).sum(dim=1).mean()
</code></pre>
<p>This works for a minibatch. Can I use it in that form? or do I have to make a class out of it and add those things:     <span class="mention">@weak_script_method</span>  <span class="mention">@weak_module</span> and inherit from somehere?</p>

username: KFrank
cooked: <p>Hello John!</p>
<aside class="quote no-group" data-username="John_Deterious" data-post="1" data-topic="50871" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_deterious/48/13825_2.png" class="avatar"> John_Deterious:</div>
<blockquote>
<p>And I failed to find what I wanted. I basically have multiclass problem, and the targets for each instance is a vector of soft classes.</p>
</blockquote>
</aside>
<p>I believe that you are correct.  As far as I am aware, all of the<br>
pre-packaged pytorch cross-entropy loss functions take class<br>
labels for their targets, rather than probability distributions<br>
across the classes.</p>
<aside class="quote no-group">
<blockquote>
<p>To be concrete:<br>
nueral net output [0.1, 0.5, 0.4]<br>
correct label [0.2, 0.4, 0.4]</p>
</blockquote>
</aside>
<p>Looking at your numbers, it appears that both your predictions<br>
(neural-network output) and your targets (“correct label”) are<br>
probability distributions across the classes.  This makes sense.</p>
<p>(One imagines that your predictions are the output of a softmax<br>
layer.)</p>
<aside class="quote no-group">
<blockquote>
<p>I created my simple Python function to do it, but can anyone please help me wrap it as a proper pytorch loss function?</p>
<pre><code class="lang-auto">&gt; def CXE(predicted, target):
&gt;     return -(target * t.log(predicted)).sum(dim=1).mean()
</code></pre>
<p>This works for a minibatch. Can I use it in that form? or do I have to make a class out of it and add those things:     <span class="mention">@weak_script_method</span>  <span class="mention">@weak_module</span> and inherit from somehere?</p>
</blockquote>
</aside>
<p>You can use your <code>CXE</code> function just as it stands.  There is no<br>
requirement (or even any particular reason) to wrap it some kind<br>
of “official” pytorch loss function.  Because you are using standard<br>
pytorch tensor functions to do your calculations, autograd will<br>
work correctly without you doing anything special.</p>
<p>So you should be able to run something like:</p>
<pre data-code-wrap="python"><code class="lang-python">opt.zero_grad()
predicted = model (input)
loss = CXE (predicted, target)
loss.backward()
opt.step()
</code></pre>
<p>(Here, <code>input</code>, <code>predicted</code>, and <code>target</code> are understood as being<br>
for a batch of samples.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: John_Deterious
cooked: <p>That is the most well-written and thorough answer I’ve seen in a while. Hats off. Many thanks Frank.</p>