Binary classification

bigtree: <p>Hi<br>
I am trying to build a (people and background) segmentation network.<br>
I have a working network but for distance estimation. I plan to change this a little bit for the segmetation woor.<br>
Here is my distance model:</p>
<pre><code class="lang-auto">
class network(nn.Module):
    distance estimation layers
                     ...
    distance estimation layers
       self.scratch.output_conv = nn.Sequential(
            nn.Conv2d(features, features//2, kernel_size=3, stride=1, padding=1, groups=self.groups),
            Interpolate(scale_factor=2, mode="bilinear"),
            nn.Conv2d(features//2, 32, kernel_size=3, stride=1, padding=1),
            self.scratch.activation,
            nn.Conv2d(32, 1, kernel_size=1, stride=1, padding=0),
            nn.ReLU(True) if non_negative else nn.Identity(),
            nn.Identity(),
        )
       def forward(self, x):
            distance estimation layers
                      .....
             output = self.scratch.output_conv(...)
            return output
</code></pre>
<p>since the output is a dense map.<br>
So to output either 0 or 1 for each pixel, I made the following change (Add Sigmoid and binary the output):</p>
<pre><code class="lang-auto">class network(nn.Module):
    distance estimation layers
                     ...
    distance estimation layers
       self.scratch.output_conv = nn.Sequential(
            nn.Conv2d(features, features//2, kernel_size=3, stride=1, padding=1, groups=self.groups),
            Interpolate(scale_factor=2, mode="bilinear"),
            nn.Conv2d(features//2, 32, kernel_size=3, stride=1, padding=1),
            self.scratch.activation,
            nn.Conv2d(32, 1, kernel_size=1, stride=1, padding=0),
            nn.ReLU(True) if non_negative else nn.Identity(),
            nn.Identity(),
            nn.Sigmoid()
        )
       def forward(self, x):
            distance estimation layers
                      .....
             output = self.scratch.output_conv(...)
            pred = out &gt; 0.75
            pred = pred * 1.0
        return pred 
</code></pre>
<p>During training, I use the loss function:</p>
<pre><code class="lang-auto">loss_f= nn.BCELoss()
est= model(image)
loss = loss_f(est, target)
loss.backward()
</code></pre>
<p>When run I got</p>
<pre><code class="lang-auto">  loss.backward()
  File "/home/bigtree/miniconda3/envs/bigtree/lib/python3.7/site-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/bigtree/miniconda3/envs/bigtree/lib/python3.7/site-packages/torch/autograd/__init__.py", line 147, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Please let me know if I did something wrong.<br>
Thanks</p>

KFrank: <p>Hi Bigtree!</p>
<aside class="quote no-group quote-modified" data-username="bigtree" data-post="1" data-topic="123941" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bigtree/48/42447_2.png" class="avatar"> bigtree:</div>
<blockquote>
<p>So to output either 0 or 1 for each pixel, I made the following change (Add Sigmoid and binary the output):</p>
<pre><code class="lang-auto">             output = self.scratch.output_conv(...)
            pred = out &gt; 0.75
</code></pre>
</blockquote>
</aside>
<p>This is your problem – the thresholding step is not usefully<br>
differentiable, so it breaks the computation graph, leading<br>
to your error.</p>
<p>It is perfectly appropriate (and in fact necessary) to train with<br>
continuous predictions that have not been thresholded (or otherwise<br>
converted to a discrete result).</p>
<aside class="quote no-group">
<blockquote>
<p>During training, I use the loss function:</p>
<pre><code class="lang-auto">loss_f= nn.BCELoss()
</code></pre>
</blockquote>
</aside>
<p>As an aside, you a better off, for reasons of numerical stability,<br>
using <code>BCEWithLogitsLoss</code> (and no final <code>Sigmoid</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

bigtree: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
Thank you very much for your prompt reply.<br>
Then how about the inference.<br>
Without Sigmoid, the out will be a random floating number may out of range [0, 1].<br>
what should do to regulate the out to either 0 or 1 for each pixel when doing inference?</p>
<p>Thanks,</p>

KFrank: <p>Hi Bigtree!</p>
<aside class="quote no-group" data-username="bigtree" data-post="3" data-topic="123941" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bigtree/48/42447_2.png" class="avatar"> bigtree:</div>
<blockquote>
<p>Without Sigmoid, the out will be a random floating number may out of range [0, 1].<br>
what should do to regulate the out to either 0 or 1 for each pixel when doing inference?</p>
</blockquote>
</aside>
<p>Well, it won’t be a <em>random</em> floating-point number.  It will still be the<br>
prediction of the network, but expressed as a raw-score <em>logit</em> that<br>
runs from <code>-inf</code> to <code>inf</code>, rather than a probability between 0 and 1.</p>
<p>The question is how to convert a probabilistic prediction (whether<br>
expressed as a logit or a probability) to a hard 0-1 prediction.</p>
<p>If it were a probability, we would typically threshold the probability<br>
against 0.5 (e.g., <code>p &lt; 0.5</code> means a hard prediction of <code>0</code>, while<br>
<code>p &gt; 0.5</code> means a hard prediction of <code>1</code>).  This is equivalent to<br>
thresholding the logit against 0.0:  <code>logit &lt; 0.0</code> means <code>0</code> and<br>
<code>logit &gt; 0.0</code> means <code>1</code>.  This is because a <code>logit</code> of <code>0.0</code><br>
corresponds to a probability of <code>0.5</code>, i.e. <code>sigmoid (0.0) = 0.5</code>.</p>
<p>In your case:</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">pred = out &gt; 0.75
</code></pre>
</blockquote>
</aside>
<p>you choose to threshold your probability against <code>0.75</code>.  (There’s<br>
nothing wrong with doing this.)  This corresponds to thresholding<br>
your logit against <code>log (0.75 / (1 - 0.75)) = 1.099</code> (which<br>
is to say that <code>sigmoid (1.099) = 0.75</code>).</p>
<p>That is, leave out the <code>Sigmoid</code> (so that <code>out</code> is a logit), and calculate<br>
your hard 0-1 prediction as</p>
<pre><code class="lang-auto">pred = out &gt; log (0.75 / (1 - 0.75))
</code></pre>
<p>Equivalently you could pass <code>out</code> through <code>sigmoid()</code> <em>after</em><br>
calculating your loss with <code>BCEWithLogitsLoss</code>, and threshold<br>
the result against <code>0.75</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">pred = sigmoid (out) &gt; 0.75
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

bigtree: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
Yes, this works.<br>
Thank you very much for the explanation.<br>
Really appreciate that.</p>