Loss function for Floating targets

username: Shuvayan
cooked: <p>So , i have been trying to implement the distilled model concept by Hinton et. al. in the paper <a href="https://arxiv.org/abs/1503.02531" rel="nofollow noopener">Hinton Dark Knowledge</a>.</p>
<p>Accordingly i trained a cumbersome model , and depending on the results on the cumbersome model, i have to train the smaller model to fit the data.</p>
<p>Now the output from the big cumbersome model is of the shape <code>( batch_size , outputs )</code> which is same as the as the output size from the small model.</p>
<p>Now in the paper, it is mentioned to take a cross-entropy loss between the outputs from the big model which will act as soft targets for the small model. However neither the size nor the data type ( FloatTensor ) is suitable for cross-entropy loss in the pytorch library . I think there’s api available for the same in tensorflow.</p>
<p>I have tried <code>KL Divergence</code> loss for the loss between outputs of the big model and the small one and the <code>CrossEntropyLoss</code> for the outputs of the small model and the actual targets ( hard labels), but it gave me very poor generalizing errors, it seems to overfit very much.</p>
<p>How can i implement a custom loss function for the outputs from the big model ( soft targets ) and the outputs from the big model which is of the same size .<br>
or</p>
<pre><code class="lang-auto">output = torch.rand( 2 ,4 ) 
soft_targets = torch.rand( 2 , 4 ) 
labels = torch.rand( 2 ,1).squeeze()
loss = criterion1( output , soft_targets )  + criterion2( output , labels ) 

</code></pre>
<p>I think criterion2 can be CrossEntropyLoss, what can be criterion1 in this case, please don’t suggest <code>KL Divergence</code> .</p>

username: KFrank
cooked: <p>Hi Shuvayan!</p>
<aside class="quote no-group" data-username="Shuvayan" data-post="1" data-topic="88847" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shuvayan/48/26602_2.png" class="avatar"> Shuvayan:</div>
<blockquote>
<p>Now in the paper, it is mentioned to take a cross-entropy loss between the outputs from the big model which will act as soft targets for the small model. However neither the size nor the data type ( FloatTensor ) is suitable for cross-entropy loss in the pytorch library . I think there’s api available for the same in tensorflow.</p>
</blockquote>
</aside>
<p>Yes, pytorch’s <code>cross_entropy_loss()</code> is a special case of cross-entropy<br>
that requires integer categorical labels (“hard targets”) for its targets.<br>
(It also takes logits, rather than probabilities, for its predictions.)</p>
<p>It does sound like you want a general cross-entropy loss that takes<br>
probabilities (“soft tagets”) for its targets.  This general version is not<br>
built in to pytorch.</p>
<p>But you can implement the general version using pytorch tensor<br>
operations.  See this earlier thread:</p>
<aside class="quote" data-post="1" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulslicer/48/9454_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501/">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    <a href="https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits" rel="noopener nofollow ugc">softmax_cross_entropy_with_logits</a> 
TF supports not needing to have hard labels for cross entropy loss: 
logits = [[4.0, 2.0, 1.0], [0.0, 5.0, 1.0]] labels = [[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]] tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=logits) 
Can we do the same thing in Pytorch?
  </blockquote>
</aside>

<p>Note that the <code>softXEnt()</code> implemented in this post also takes logits<br>
for its predictions.  If your use case requires you to pass in probabilities<br>
for your predictions (less numerically stable), you will have to modify<br>
<code>softXEnt()</code> accordingly.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Shuvayan
cooked: <p>Yes, i think this might solve my problem.<br>
I have to pass direct probabilies generated from a  model ( the big one ) and after doing softmax on it to the loss function , where both are of the same size .</p>
<pre><code class="lang-auto">&gt;&gt;&gt; def softXEnt (input, target):
...     return  -(target * input).sum() / input.shape[0]
...
&gt;&gt;&gt; input = torch.rand( 2, 5 )
&gt;&gt;&gt; target = torch.rand( 2, 5 )
&gt;&gt;&gt; input = torch.nn.functional.log_softmax( input , dim =1 )
&gt;&gt;&gt; target = torch.nn.functional.softmax( target , dim =1 )
&gt;&gt;&gt; softXEnt( input , target )
tensor(1.6736)
&gt;&gt;&gt; torch.nn.KLDivLoss(reduction='batchmean')(input ,target )
tensor(0.0981)
</code></pre>
<p>Since the results are different from KL divergence loss i think it might be actually helpful for my use case . Thanks .</p>