Loss function for predicting a distribution?

MartinLwx: <p>Hi guys. Let’s say I want to predict three value(they add up to <code>1</code>). The labels are also three values(they add up to <code>1</code>).</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> I regard it as a distribution learning problem(not sure if I am right). so I am currently using the <code>F.kl_div</code> as my loss function. But I found that I can also use the <code>MAE</code> for each output to calculate the loss(a simple regression problem with 3 head). The experimental results are similar.</p>
<pre><code class="lang-python">&gt;&gt;&gt; logits        # the output of my model
tensor([-0.1398, -0.1189, -0.9538])
&gt;&gt;&gt; target
tensor([0.1500, 0.2500, 0.6000])
# using kl_dev
&gt;&gt;&gt; F.kl_div(F.log_softmax(logits.unsqueeze(0), dim=1), target.unsqueeze(0), reduction='batchmean')
tensor(0.4474)
# using MAE
&gt;&gt;&gt; F.l1_loss(F.softmax(logits.unsqueeze(0), dim=1), target.unsqueeze(0), reduction='mean')
tensor(0.2801)
</code></pre>
<p>I wonder if I did it right? Which loss function should I use for this kind of problem. Or is there any other correct way to do this?</p>
<p>Any help will be appreciated : )</p>

ParGG: <p>From the optimization problem prospective, little would change. Both will try to match your ground truth (Beware that L1 Norm is ok but L2 not). The difference lies in the interpretation that you give to the values of the loss and how they optimise your model.</p>
<p>Keeping it short. Concerning the interpretation, KL Divergence gives you the amount of information lost when your logits are used to approximate your target (one of the many interpretations). Which is much more meaningful than the L1 distance, that gives you the mean error that you commit when approximating the target with the logits. The two are somewhat related by this: <a href="https://en.wikipedia.org/wiki/Total_variation_distance_of_probability_measures" class="inline-onebox" rel="noopener nofollow ugc">Total variation distance of probability measures - Wikipedia</a>.</p>

MartinLwx: <p>Thanks for your reply, it helped me a lot.</p>
<p>I notice that the KL Divergence isn’t symmetric, which makes me wonder if it would be better to use JS Divergence to optimize the model. <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_spiral_eyes.png?v=12" title=":face_with_spiral_eyes:" class="emoji" alt=":face_with_spiral_eyes:" loading="lazy" width="20" height="20"></p>
<p>Have a good day. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

ParGG: <p>You can read this about the topic and pick the best for your need <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.quora.com/Why-isnt-the-Jensen-Shannon-divergence-used-more-often-than-the-Kullback-Leibler-since-JS-is-symmetric-thus-possibly-a-better-indicator-of-distance">
  <header class="source">
      <img src="https://qsf.cf2.quoracdn.net/-4-images.favicon-new.ico-26-07ecf7cd341b6919.ico" class="site-icon" width="256" height="256">

      <a href="https://www.quora.com/Why-isnt-the-Jensen-Shannon-divergence-used-more-often-than-the-Kullback-Leibler-since-JS-is-symmetric-thus-possibly-a-better-indicator-of-distance" target="_blank" rel="noopener nofollow ugc">Quora</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:600/315;"><img src="https://qph.cf2.quoracdn.net/main-custom-t-980-600x315-sqxxospfciejqkjfytssmfbdcvzbbxla.jpeg" class="thumbnail" width="600" height="315"></div>

<h3><a href="https://www.quora.com/Why-isnt-the-Jensen-Shannon-divergence-used-more-often-than-the-Kullback-Leibler-since-JS-is-symmetric-thus-possibly-a-better-indicator-of-distance" target="_blank" rel="noopener nofollow ugc">Why isn't the Jensen-Shannon divergence used more often than the...</a></h3>

  <p>Answer (1 of 4): The Kullback-Leibler divergence has a few nice properties, one of them being that KL[q;p] kind of abhors regions where q(x) have non-null mass and p(x) has null mass. This might look like a bug, but it’s actually a feature in certain...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


MartinLwx: <p>Thanks for your advice <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>