Looking for loss function to enforce binary (near 0 or 1) output

username: metro
cooked: <p>Hi there,<br>
I’m looking for a loss function which will motivate a network to output on the last layer either near 0 or near 1 values. I’m imagining a function that looks similar to this. The output layer is of type float/double and will be used as a mask.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e03996781716dc645c8647d50453b3d3826f5ffd.png" alt="loss" data-base62-sha1="vZAkKeBDXL5UQWrgFf8sEd9oNgF" width="467" height="330"></p>
<p>Background:<br>
I’m training an unsupervised network to predict masks. My evaluation function would ideally get near 0 or 1 values. It can handle anything, but it would make the most sense if it was 0 or 1. I’m planning to sum the loss function, which will motivate 0/1 values, with the loss of my evaluation function. While clamping to [0, 1] would solve the left and right side, it would not help with values in between.</p>
<p>I cannot use the Binary Cross Entropy loss because I do not have the target values.</p>
<p>While I could use a combination of torch.gt and clamping I don’t know if this would work as well as it would allow the model to be “uncertain” i.e. output values which are extremely close to 0.5 e.g. 0.499999999 or 0.50000000001. Please tell me if my reasoning here is flawed.</p>
<p>If anyone has any better ideas on how to solve the 0/1 mask problem other than with such a loss function, I’m all open ears!</p>
<p>Cheers!</p>

username: KFrank
cooked: <p>Hi Metro!</p>
<aside class="quote no-group" data-username="metro" data-post="1" data-topic="124889" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/metro/48/29729_2.png" class="avatar"> metro:</div>
<blockquote>
<p>I’m looking for a loss function which will motivate a network to output on the last layer either near 0 or near 1 values.</p>
</blockquote>
</aside>
<p>I don’t understand your use case, but, in general, you can certainly<br>
penalize outputs that are not close to 0 or 1.</p>
<p>Let <code>x</code> be a single output.  You could then try things like:</p>
<pre data-code-wrap="python"><code class="lang-python">penalty_0_1 = (x * (1.0 - x))**2
penalty_0_1 = torch.abs (x * (1.0 - x))
penalty_0_1 = ((x * (1.0 - x))**2)**alpha
</code></pre>
<p>where the last version includes the parameter <code>alpha</code> that can be<br>
used to tune the shape of the “wells” at 0 and 1.</p>
<p>(<code>x</code> would likely be a pytorch tensor, so you would typically sum<br>
after calculating the penalty to get a single scalar penalty that would<br>
push the individual elements of the tensor towards 0 or 1.)</p>
<aside class="quote no-group">
<blockquote>
<p>it would allow the model to be “uncertain” i.e. output values which are extremely close to 0.5 e.g. 0.499999999 or 0.50000000001.</p>
</blockquote>
</aside>
<p>There is no (reasonable) way to avoid having an “uncertain” point<br>
somewhere between 0 and 1 (most likely at 0.5).  But more importantly,<br>
you don’t want the penalty at 0.5 to be so large that it is impractical<br>
for the network output to move from near 0 to near 1 or vice versa.</p>
<p>Suppose that the network output “should” be (near) 1, but happens<br>
to start out near 0.  As the network trains, you want it to be able to<br>
get over the penalty hump, and not stay stuck near 0 forever.  (One<br>
approach to this issue is to multiply your penalty with a weight.  At<br>
the beginning of training start with a small weight so that the network<br>
output can move easily between 0 and 1, but increase the weight as<br>
training progresses so that the network output is forced increasingly<br>
close to 0 or 1.)</p>
<p>Best.</p>
<p>K. Frank</p>