Model always predicts tensors of 0 values

niloleart: <p>Hi,</p>
<p>I’m trying to solve a classification problem with 2 classes for a biomedical application using images.<br>
After reading a lot in this forum I’ve seen one option is using BCEWithLogitsLoss and one single output neuron. I have class imbalance, so I am using pos_weight parameter, although I don’t know if correctly.<br>
I have some questions…</p>
<ol>
<li>Using Adam optimizer is a good idea? Or should I use SGD?</li>
</ol>
<pre><code class="lang-auto">optimizer_ft = optim.Adam(params_to_update, hparams['learning_rate'])

num_positives = torch.tensor(sum(labels == 1), dtype=float)  #  250
num_negatives = torch.tensor(len(labels) - num_positives, dtype=float)  #  604
pos_weight = (num_negatives / num_positives) # around 2.4

criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight).to(get_device())
</code></pre>
<p>This is my train function:<br>
2) Is it correct?</p>
<pre><code class="lang-auto"> for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch + 1, num_epochs))
        print('-' * 10)

        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()
            else:
                model.eval()

            running_loss = 0.0
            running_corrects = 0

            for inputs, labels_raw, _, _ in dataloaders[phase]:
                inputs = inputs.float()
                labels = labels_raw.float()

                inputs = inputs.to(device)
                labels = labels.to(device)

                optimizer.zero_grad()

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    loss = criterion(outputs, labels.unsqueeze(1))
                    _, preds = torch.max(outputs, dim=1)

                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == torch.argmax(labels)).item()

            epoch_loss = running_loss / len(dataloaders[phase].dataset)
            epoch_acc = running_corrects / len(dataloaders[phase].dataset)

            if phase == 'val' and epoch_acc &gt; best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())
            if phase == 'val':
                val_acc_history.append(epoch_acc)

            if phase == 'train':
                train_loss.append(epoch_loss)
                train_acc.append(epoch_acc)
            else:
                val_loss.append(epoch_loss)
                val_acc.append(epoch_acc)

            print('{} loss: {:.4f}, Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))
</code></pre>
<p>This are always my predictions:</p>
<pre><code class="lang-auto">tensor([0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')
</code></pre>
<p>This is a sample target for one batch of 8:</p>
<pre><code class="lang-auto">tensor([0., 0., 0., 1., 0., 0., 0., 1.], dtype=torch.float64)
</code></pre>
<p>There is definitely something wrong with my code but I am unable to find it. Some help will be appreciated.</p>
<p>I am also normalizing my images computing mean and std of the whole dataset for each channel and then normalizing each data split.<br>
3) Should I normalize the validation/test split with the same parameters?</p>
<p>Thanks</p>

KFrank: <p>Hi Nil!</p>
<aside class="quote no-group" data-username="niloleart" data-post="1" data-topic="167947" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/niloleart/48/46263_2.png" class="avatar"> niloleart:</div>
<blockquote>
<p>one option is using BCEWithLogitsLoss and one single output neuron.</p>
</blockquote>
</aside>
<p>With one single output neuron (if I understand correctly what you mean), the<br>
last dimension of your <code>outputs</code> will have size 1, e.g., <code>outputs</code> would have a<br>
shape of something like <code>[nBatch, 1]</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">                    _, preds = torch.max(outputs, dim=1)
...
                running_corrects += torch.sum(preds == torch.argmax(labels)).item()
</code></pre>
<p>This are always my predictions:</p>
<pre><code class="lang-auto">tensor([0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')
</code></pre>
</blockquote>
</aside>
<p>Your problem (assuming that your “predictions” is your <code>preds</code>) is that<br>
<code>_, preds = torch.max(outputs, dim=1)</code> will <em>always</em> return <code>0</code> because<br>
<code>dim = 1</code> of <code>outputs</code> has size 1.   Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; model = torch.nn.Linear (10, 1)
&gt;&gt;&gt; inputs = torch.randn (5, 10)   # batch size of 5
&gt;&gt;&gt; outputs = model (inputs)
&gt;&gt;&gt; outputs   # okay
tensor([[ 1.0431],
        [-0.8160],
        [-1.3349],
        [-0.3153],
        [ 0.9312]], grad_fn=&lt;AddmmBackward0&gt;)
&gt;&gt;&gt; _, preds = torch.max (outputs, dim = 1)   # computes argmax along dim = 1
&gt;&gt;&gt; preds   # but argmax is always 0
tensor([0, 0, 0, 0, 0])
</code></pre>
<p>For the binary case where you have only a single output “neuron” (that has<br>
not been passed through a <code>sigmoid()</code>), you would want to compute your<br>
integral yes-no predictions by thresholding <code>outputs</code> against zero:</p>
<pre data-code-wrap="python"><code class="lang-python">preds = outputs &gt; 0.0            # returns a bool
# or, if you prefer
preds = (outputs &gt; 0.0).long()   # explicit cast to long
</code></pre>
<p>(You have the analogous error in your expression for <code>running_corrects</code>,<br>
as well.)</p>
<p>Best.</p>
<p>K. Frank</p>

niloleart: <p>Thanks for the reply,</p>
<p>you are right!</p>
<p>But, as explained in the documentation, <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>:</p>
<blockquote>
<p>This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.</p>
</blockquote>
<p>So I guess my threshold should be</p>
<pre><code class="lang-auto">outputs &gt; 0.5
</code></pre>
<p>I also do not undersand how to compute the pos_weight parameter of the loss.<br>
Does it need to be computed on the whole dataset and pass a single number ?<br>
<code>pos_weight = negative_samples / positive_samples</code></p>
<p>or it has to be computed for every batch?</p>
<pre><code class="lang-auto">pos_weight = (labels == 0).sum() / labels.sum()
</code></pre>

KFrank: <p>Hi Nil!</p>
<aside class="quote no-group quote-modified" data-username="niloleart" data-post="3" data-topic="167947" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/niloleart/48/46263_2.png" class="avatar"> niloleart:</div>
<blockquote>
<blockquote>
<p>This loss combines a Sigmoid layer and the BCELoss in one single class.</p>
</blockquote>
<p>So I guess my threshold should be</p>
<pre><code class="lang-auto">outputs &gt; 0.5
</code></pre>
</blockquote>
</aside>
<p>No. You are right that <code>BCWithLogitsloss</code> computes <code>sigmoid()</code> internally<br>
(actually <code>log_sigmoid()</code>), but this doesn’t affect your <code>outputs</code>.  (It affects<br>
how <code>outputs</code> is used in the loss computation, but doesn’t modify <code>outputs</code>.)<br>
They will still be logits that range from <code>-inf</code> to <code>inf</code>, so you should still<br>
threshold against <code>0.0</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Does it need to be computed on the whole dataset and pass a single number ?<br>
<code>pos_weight = negative_samples / positive_samples</code></p>
<p>or it has to be computed for every batch?</p>
<pre><code class="lang-auto">pos_weight = (labels == 0).sum() / labels.sum()
</code></pre>
</blockquote>
</aside>
<p>First, your imbalance of about 2.4 is not really very large, so you probably<br>
don’t need <code>pos_weight</code> in practice.  (It’s fine to use it though, and could<br>
offer some minor benefits.)</p>
<p>Second, the <em>exact</em> value of <code>pos_weight</code> shouldn’t matter.  If your training<br>
is sensitive to small changes is <code>pos_weight</code> you probably have some sort<br>
of instability in your training that you would also see if you varied some<br>
other hyperparameter or used different random initialization of your network.</p>
<p>So it shouldn’t really matter exactly how you compute <code>pos_weight</code> (as long<br>
as you don’t end up with <code>inf</code>).  I would generally recommend computing<br>
<code>pos_weight</code> for your entire training set, or if that is inconvenient, for a<br>
representative subset of your training set.</p>
<p>(Note, your second expression, <code>(labels == 0).sum() / labels.sum()</code>,<br>
should be something like <code>(labels == 0).sum() / (labels == 1).sum()</code>.)</p>
<p>(One could plausibly argue that there is a mild theoretical justification for<br>
computing <code>pos_weight</code> on a per-batch basis, but it really shouldn’t matter.)</p>
<p>Best.</p>
<p>K. Frank</p>