Sigmoid + BCELoss not similar to BCEwithLogitsLOSS?

raouf_ks: <p>hello everyone ,<br>
I’m coding the dropout as a variational inference as [ Gal et al 2016 ] explain it</p>
<p>I have a problem regarding “BCE With Logits Loss”</p>
<p>My simple neural net is</p>
<pre data-code-wrap="class"><code class="lang-class">    def __init__(self, hidden_size, input_size=1, dropout_p=0.25 ):
        super().__init__()
        self.dropout_p = dropout_p
        
        self.hidden1 = nn.Linear(input_size,hidden_size)
        self.relu    = nn.ReLU()
        self.output  = nn.Linear(hidden_size,1)

    def forward(self, x):
        x = self.relu(self.hidden1(x))
        x = F.dropout(x,p=self.dropout_p)
        x = self.output(x)
        x = torch.sigmoid(x)
        return x

bnn = MLP(hidden_size=20,input_size=2)
bnn.train()
criterion = nn.BCELoss()
optimizer_bnn = torch.optim.SGD(bnn.parameters(), lr=0.01, momentum=0.90, nesterov=True, weight_decay=1e-6)
</code></pre>
<p>my training loop</p>
<pre><code class="lang-auto">torch.manual_seed(7)

fig, ax = plt.subplots(figsize=(7,7))
num_epochs=200

# To Do here

#########################  classical Gradient training loop ################################# 
for epoch in range(num_epochs):
    for (X,Y) in train_dataloader:  
            # Forward pass
            outputs = bnn(X.float())
            loss = criterion(outputs.float().reshape(-1), Y.float())

            # Backward and optimize
            optimizer_bnn.zero_grad()
            loss.backward()
            optimizer_bnn.step()
            
    # For plotting and showing learning process at each epoch, uncomment line below
    if (epoch+1)%10==0:
        plot_decision_boundary( bnn, X, Y, epoch, ((outputs.squeeze()&gt;=0.5) == Y).float().mean(), 
                                nbh=4, model_type='mcdropout')
##############################################################################################



print('Finished Training')
</code></pre>
<p>My plot function</p>
<pre><code class="lang-auto"># Useful function: plot and show learning process in classification
def plot_decision_boundary(model, X, Y, epoch, accuracy, model_type='classic', samples=100, nbh=2, cmap='RdBu'):    
    h = 0.02*nbh
    x_min, x_max = X[:,0].min() - 10*h, X[:,0].max() + 10*h
    y_min, y_max = X[:,1].min() - 10*h, X[:,1].max() + 10*h
    xx, yy = np.meshgrid(np.arange(x_min*2, x_max*2, h),
                         np.arange(y_min*2, y_max*2, h))
    
    test_tensor = torch.from_numpy(np.c_[xx.ravel(), yy.ravel()]).float()
    if model_type=='classic':
        model.eval()
        pred = model(test_tensor)
    elif model_type=='svi':
        pred = model.forward(test_tensor, n_samples=samples).mean(0)
    elif model_type=='mcdropout':
        model.eval()
        model.training = True
        outputs = torch.zeros(samples, test_tensor.shape[0], 1)
        for i in range(samples):
            outputs[i] = model(test_tensor)
        pred = outputs.mean(0).squeeze()
    Z = pred.reshape(xx.shape).detach().numpy()

    plt.cla()
    ax.set_title('Classification Analysis')
    ax.contourf(xx, yy, Z, cmap=cmap, alpha=0.25)
    ax.contour(xx, yy, Z, colors='k', linestyles=':', linewidths=0.7)
    ax.scatter(X[:,0], X[:,1], c=Y, cmap='Paired_r', edgecolors='k');
    ax.text(-4, -7, f'Epoch = {epoch+1}, Accuracy = {accuracy:.2%}', fontdict={'size': 12, 'fontweight': 'bold'})
    display.display(plt.gcf())
    display.clear_output(wait=True)
</code></pre>
<p>My problem is if I keep the sigmoid in the forward and use BCELoss I get this final result<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/828e6eb53c52d0262c16c8d779a7fc4101c513b2.png" alt="Screenshot from 2020-01-27 23-04-20" data-base62-sha1="iCXel2POQcAmtvGetakQzOxOlma" width="510" height="489"><br>
However if I change it without a sigmoid function and use BCEwithLogits I get this result<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/7/2762acb0546814b2867402c53832954ab780c751.png" alt="Screenshot from 2020-01-27 23-17-08" data-base62-sha1="5Cq1TJO9J3l0xk0g19Ag7JFaWFr" width="527" height="473"></p>

KFrank: <p>Hi Raouf!</p>
<p>I don’t follow in detail what you are doing, but be aware that<br>
the output of a model with a final <code>sigmoid()</code> layer will, of<br>
course, be different than the output of the analogous model<br>
that lacks that layer.</p>
<p>With the <code>sigmoid()</code> layer your model returns <em>probabilities.</em><br>
Without the <code>sigmoid()</code> layer, your model returns “raw scores”<br>
that are called <em>logits.</em></p>
<p>Some comments appear in line, below.</p>
<aside class="quote no-group quote-modified" data-username="raouf_ks" data-post="1" data-topic="67826" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raouf_ks/48/11109_2.png" class="avatar"> raouf_ks:</div>
<blockquote>
<p>I have a problem regarding “BCE With Logits Loss”</p>
<p>My simple neural net is</p>
<pre><code class="lang-auto">        x = self.output(x)
        x = torch.sigmoid(x)
        return x
...
</code></pre>
</blockquote>
</aside>
<p>Removing the <code>x = torch.sigmoid(x)</code> will, of course, change<br>
the value <code>x</code> that your model returns.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">criterion = nn.BCELoss()
</code></pre>
</blockquote>
</aside>
<p>The <em>loss</em> you calculate will be (more or less) the same if you<br>
then use <code>criterion = nn.BCEWithLogitsLoss()</code> without<br>
the <code>sigmoid()</code> layer, but the model’s output is different.  This<br>
difference – logits instead of probabilities – is compensated for<br>
by using the different loss function.</p>
<aside class="quote no-group">
<blockquote>
<p>my training loop</p>
<pre><code class="lang-auto">...
            outputs = bnn(X.float())
            loss = criterion(outputs.float().reshape(-1), Y.float())
...
        plot_decision_boundary( bnn, X, Y, epoch, ((outputs.squeeze()&gt;=0.5) == Y).float().mean(), 
                                nbh=4, model_type='mcdropout')
</code></pre>
</blockquote>
</aside>
<p>Using <code>outputs.squeeze()&gt;=0.5</code> as your binary predictions<br>
is appropriate when using the final <code>sigmoid()</code> layer.  But unless<br>
you change the logic for your prediction / accuracy calculation<br>
when you remove the <code>sigmoid()</code> layer you will get different<br>
(and incorrect) results.</p>
<p>When <code>outputs</code> are the logits you get without the <code>sigmoid()</code><br>
layer, you should use <code>outputs.squeeze()&gt;=0.0</code> to get your<br>
binary predictions.</p>
<aside class="quote no-group">
<blockquote>
<p>My plot function</p>
<pre><code class="lang-auto">def plot_decision_boundary(model, X, Y, epoch, accuracy, model_type='classic', samples=100, nbh=2, cmap='RdBu'):    
...
    if model_type=='classic':
        model.eval()
        pred = model(test_tensor)
...
    Z = pred.reshape(xx.shape).detach().numpy()

    plt.cla()
    ax.set_title('Classification Analysis')
    ax.contourf(xx, yy, Z, cmap=cmap, alpha=0.25)
    ax.contour(xx, yy, Z, colors='k', linestyles=':', linewidths=0.7)
</code></pre>
<p>My problem is if I keep the sigmoid in the forward and use BCELoss I get this final result<br>
…<br>
However if I change it without a sigmoid function and use BCEwithLogits I get this result<br>
…</p>
</blockquote>
</aside>
<p>You plot <code>Z</code>.  <code>Z</code> comes from <code>preds</code>, which comes from your model.<br>
When you remove the <code>sigmoid()</code> from your model, you change<br>
<code>preds</code> and <code>Z</code>, so your plots come out different because you are<br>
plotting something different.</p>
<p>Best.</p>
<p>K. Frank</p>

raouf_ks: <p>My mistake was very obvious !<br>
That’s the effect of coding at 2.00 am and copying my training loop from an above cell<br>
Thanks a lot Sir</p>