Loss function for log-scoring probabilistic forecasts

peter_m: <p>I’m completely new to PyTorch, so apologies in advance if this is very silly.<br>
My goal is to post-process (evolutions of) probabilistic forecasts for which the outcome is known. E.g. I have forecasts of the form <code>[.5, .6, .8]</code> and the information that this forecast resolved positively, i.e. <code>resolution=1</code>. (In this example, the forecast started off as uncertain with 50% and converged towards 80%, so not too bad.)</p>
<p>The input to my network is a probability <code>x_1</code> and more relevant information <code>x_2,x_3,...</code> accessible at the time this forecast was made. Its output should be a different probability <code>y_hat</code>. The loss function I’m interested in is <code>y_true*log(y_hat) + (1-y_true)*log(1-y_hat)</code>.</p>
<p>Since <code>y_true</code> is either 0 or 1, this can be seen as the Kullback-Leibler divergence loss if we consider not only the KL distance between <code>y_true</code> and <code>y_hat</code>, but also add the KL distance between <code>1-y_true</code> and <code>1-y_hat</code>.</p>
<p>(I’d also like to weigh these contributions to the log score by the time elapsed between to consecutive features, but unlike other losses, <a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html#torch.nn.KLDivLoss" class="inline-onebox" rel="noopener nofollow ugc">KLDivLoss — PyTorch 1.11.0 documentation</a> doesn’t seem to take a <code>weight</code> argument.)</p>
<p>Is there a neat way to get the loss function I described above by somehow mirroring output data or do I have to write it “from scratch” myself?</p>

KFrank: <p>Hi Peter!</p>
<aside class="quote no-group" data-username="peter_m" data-post="1" data-topic="154724" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_m/48/50739_2.png" class="avatar"> peter_m:</div>
<blockquote>
<p>The loss function I’m interested in is <code>y_true*log(y_hat) + (1-y_true)*log(1-y_hat)</code>.</p>
</blockquote>
</aside>
<p>This is pytorch’s <a href="https://pytorch.org/docs/1.11/generated/torch.nn.BCELoss.html" rel="noopener nofollow ugc">BCELoss</a>.</p>
<p>However, for reasons of numerical stability, you will be better off using<br>
<a href="https://pytorch.org/docs/1.11/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>.  To do this, you will have to modify your network to<br>
predict <em>logits</em> that run from <code>-inf</code> to <code>inf</code> (rather than probabilities that run<br>
from <code>0.0</code> to <code>1.0</code>).</p>
<p>You would typically do this by having the final layer of your network be<br>
a <code>Linear</code>, <em>not</em> followed by a <code>sigmoid()</code> (that would have been used to<br>
convert logits into probabilities).</p>
<p>It’s worth noting that both <code>BCELoss</code> and <code>BCEWithLogitsLoss</code> take <code>weight</code><br>
constructor arguments.)</p>
<p>As an aside, cross entropy is related to the Kullback-Leibler divergence,<br>
although the two are not identical.</p>
<p>Best.</p>
<p>K. Frank</p>

peter_m: <p>Hi Frank,</p>
<p>That’s exactly what I was looking for, I don’t know how I overlooked it until now. Thank you very much!</p>
<p>Best,<br>
Peter</p>