BCELoss vs BCEWithLogitsLoss on Autoencoder

Giuseppe: <p>Hy all, i have a problem in the code below</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.optim import SGD, Adam
from torch.optim.lr_scheduler import StepLR
import numpy as np
from dataset import Dataset
from torch.utils.data import DataLoader
from torchnet.meter import AverageValueMeter
from torchnet.logger import VisdomPlotLogger, VisdomSaver
from nets import AutoEncoder
import torch.multiprocessing as multiprocessing



def train(model, train_loader, valid_loader, exp_name = "prova",  lr=0.000001, epochs=1000, wd = 0.000001):

  

    optimizer = Adam(params=model.parameters(),lr = lr, weight_decay=wd)
    scheduler = StepLR(optimizer, step_size=50, gamma=0.5)

    # meters
    lossAE_meter = AverageValueMeter()
    

    # device
    device = "cuda if torch.cuda.is_available() else "cpu
    model.to(device)

    loader = {"train": train_loader, "validation": valid_loader}

    loss_AE_logger = VisdomPlotLogger('line', env=exp_name, opts={'title': 'LossAE', 'legend': ['train', 'validation']})
  
    visdom_saver = VisdomSaver(envs=[exp_name])

    last_best_loss = np.inf
    for e in range(epochs):
        for mode in ["train", "validation"]:


            lossAE_meter.reset()
            
            model.train() if mode == "train" else model.eval()

            with torch.set_grad_enabled(mode == "train"):  

                for i, batch in enumerate(loader[mode]):

                    optimizer.zero_grad()
                   
                    x = batch["Input"].to(device)
                    Target = batch["Target"].to(device)

                    code, rec = model(x)
                    input = nn.Sigmoid(rec)
                    lossAE = nn.BCELoss(input, Target)
                    

                    l = lossAE
                    if mode == "train":

                        l.backward()
                        optimizer.step()

                    else:
                        if l &lt; last_best_loss:
                            torch.save(model.state_dict(), 'Best_%s.pth' % exp_name)
                            last_best_loss = l



                    n = x.shape[0]  

                    lossAE_meter.add(lossAE.item() * n, n)
       



                    if mode == "train":
                        loss_AE_logger.log(e + (i + 1) / len(loader[mode]), lossAE_meter.value()[0]*100, name=mode)
                      



            loss_AE_logger.log(e + (i + 1) / len(loader[mode]), lossAE_meter.value()[0]*100, name=mode)
           
        scheduler.step()

        #salviamo l'ambiente su visdom
        visdom_saver.save()

        # conserviamo solo l'ultimo modello sovrascrivendo i vecchi, salviamo anche il best di volta in volta
        torch.save(model.state_dict(), '%s.pth' % exp_name)



    return model
</code></pre>
<p>My model is an autoencoder that try to reconstruct a binarize image, if i pass the output of my autoencdoer to nn.Sigmoid() i receive this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/giuseppe/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "&lt;ipython-input-2-1c293c75e8af&gt;", line 1, in &lt;module&gt;
    runfile('/home/giuseppe/Scrivania/EXP/main.py', wdir='/home/giuseppe/Scrivania/EXP')
  File "/home/giuseppe/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/giuseppe/pycharm-2019.2/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/giuseppe/Scrivania/EXP/main.py", line 13, in &lt;module&gt;
    train.start_all()
  File "/home/giuseppe/Scrivania/EXP/train.py", line 140, in start_all
    model_trained= train(model, train_loader, valid_loader, exp_name="Mvm_AE", epochs=400)
  File "/home/giuseppe/Scrivania/EXP/train.py", line 75, in train
    input = nn.Sigmoid(rec)
TypeError: __init__() takes 1 positional argument but 2 were given

</code></pre>
<p>Where am i wrong?<br>
Furthermore i would like to use  BCEWithLogitsLoss, what should I pass as pos_weight?</p>

KFrank: <p>Hi Giuseppe!</p>
<aside class="quote no-group quote-modified" data-username="Giuseppe" data-post="1" data-topic="104874" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/giuseppe/48/14328_2.png" class="avatar"> Giuseppe:</div>
<blockquote>
<pre><code class="lang-auto">                    input = nn.Sigmoid(rec)
                    lossAE = nn.BCELoss(input, Target)
</code></pre>
<pre><code class="lang-auto">    input = nn.Sigmoid(rec)
TypeError: __init__() takes 1 positional argument but 2 were given

</code></pre>
</blockquote>
</aside>
<p>You are calling the <em>constructor</em> for <code>Sigmoid</code> incorrectly, namely with<br>
an explicit argument.  (There is also the hidden <code>self</code> argument.  That<br>
is why the error message complains about two arguments.)</p>
<p>In more detail, <code>torch.nn.Sigmoid</code> is a class, while <code>torch.sigmoid()</code><br>
is a function (as is the deprecated <code>torch.nn.functional.sigmoid()</code>).<br>
So you have to construct an instance of <code>Sigmoid</code> before you can call<br>
it as a so-called <em>function object.</em>  (You can construct it on the fly, if you<br>
like.)</p>
<p>Thus:</p>
<pre data-code-wrap="python"><code class="lang-python">activation = nn.Sigmoid()
...
input = activation (rec)

# or

input = nn.Sigmoid() (rec)

# or

input = torch.sigmoid (rec)
</code></pre>
<p>(All three do the same thing.)</p>
<aside class="quote no-group">
<blockquote>
<p>Furthermore i would like to use  BCEWithLogitsLoss, what should I pass as pos_weight?</p>
</blockquote>
</aside>
<p>Class weights in loss functions are typically used to account for<br>
unbalanced datasets where certain classes occur significantly more<br>
frequently than others.</p>
<p>For a binary classification problem – including a multi-label, multi-class<br>
problem, which is basically <code>nClass</code> binary problems being processed<br>
“in parallel” – you have two classes, “negative” and “positive.”</p>
<p>Common practice is to use <code>pos_weight = n_negative / n_positive</code>,<br>
passed into the constructor of <code>BCEWithLogitsLoss</code>.  (In the multi-label<br>
case, you would pass in a vector of <code>nClass</code> <code>pos_weight</code> values, one<br>
for each of your classes.)</p>
<p>(For whatever reason, <code>BCELoss</code> doesn’t take a <code>pos_weight</code><br>
constructor argument.)</p>
<p>As a side note, you are correct to prefer <code>BCEWithLogitsLoss</code>,<br>
<em>without the explicit</em> <code>Sigmoid</code>, over <code>BCELoss</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

Giuseppe: <p>One only answer: the best reply that I ever read.<br>
Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Giuseppe: <p>Which way do i have to use the input and target if i use a BCEWithLogitLoss? I think a way like this:<br>
<code>criterion = torch.nn.BCEWithLogisLoss</code><br>
<code>criterion(input, target)</code><br>
Why i have numbers different from 0 and 1?<br>
I think i have to use the sigmoid function when i infer the model, but the minimum number is 0.5 for all element <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> Why?</p>