Sigmoid Giving only Negative Outputs

Atharva_Dubey: <p>Hello All,<br>
I am building an LSTM based classifier for EEG motor imagery Data for 2 classes. The data is from a 64 channel EEG and each channel has 20000 data points. The model is being trained for 50 epochs and converges for a decent loss. While testing the model for an individual file, the Sigmoid outputs only negative value irrespective of the class. Please tell me where could I be going wrong. Sorry for a trivial question, I am relatively new to the field and recently shifted to Pytorch<br>
Attaching my code below</p>
<pre><code class="lang-auto">
class pytorch_lstm(net.Module):
    def __init__(self, features, hidden_size, sequence_length, loss_function='BCE'):
        super(pytorch_lstm, self).__init__()
        self.loss = loss_function
        self.features = features
        self.hidden_size = hidden_size
        self.seq_length = sequence_length
        self.lstm = net.LSTM(
            input_size=self.features,
            hidden_size=hidden_size,
            num_layers=1,
            batch_first=True
        )
        self.linear = net.Linear(self.hidden_size * self.seq_length, 1)

    def init_Hidden(self):
        hidden_state = torch.zeros(1, 1, self.hidden_size).cuda()
        cell_state = torch.zeros(1, 1, self.hidden_size).cuda()
        self.hidden = (hidden_state, cell_state)

    def forward(self, X):
        lstm_out, self.hidden = self.lstm(X, self.hidden)
        out = lstm_out.contiguous().view(1, -1)
        out = self.linear(out)
        return out

    def train_model(self, model, dataloader, num_epochs):
        least_loss = 10
        model.train()
        if self.loss == 'BCE':
            self.criterion = net.BCEWithLogitsLoss().cuda()
        elif self.loss == 'MSE':
            self.criterion = net.MSELoss().cuda()
        else:
            raise ValueError('PROVIDE A VALID LOSS FUNCTION')

        optimizer = torch.optim.Adam(model.parameters())
        training_loss = []
        for i in range(num_epochs):
            optimizer.zero_grad()
            st = time.time()
            epoch_loss = 0
            for _, (x, y) in enumerate(dataloader):
                model.init_Hidden()
                output = model(x)
                loss = self.criterion(output.view(-1), y)
                loss.backward()
                optimizer.step()
                print(loss.item())
                epoch_loss += loss.item()
            et = time.time()
            avg_loss = epoch_loss / 438
            print('----------------------------------------------------------')
            print('\033[1m' + 'TOTAL_TIME_PER_EPOCH = ' + str(et - st) + '\033[1m')
            print('\033[1m' + 'AVERAGE_LOSS = ' + str(avg_loss) + '\033[1m')
            training_loss.append(avg_loss)
            if avg_loss&lt;least_loss:
                torch.save(model.state_dict(), 'path' + self.loss +'.pth')
                least_loss = avg_loss

        plt.plot(training_loss)
        plt.xlabel('EPOCHS')
        plt.ylabel('LOSS')
        plt.show()
</code></pre>
<p>The model outputs something like -</p>
<pre><code class="lang-auto">tensor([[-0.0545]], device='cuda:0') CLASS 1
tensor([[-0.0541]], device='cuda:0') CLASS 2
tensor([[-0.0522]], device='cuda:0') CLASS 1
</code></pre>
<p>Interestingly, the sigmoid outputs are quite close, Can someone please tell me why ?<br>
Thanks in Advance</p>

KFrank: <p>Hi Atharva!</p>
<aside class="quote no-group quote-modified" data-username="Atharva_Dubey" data-post="1" data-topic="67903" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atharva_dubey/48/12127_2.png" class="avatar"> Atharva_Dubey:</div>
<blockquote>
<p>the Sigmoid outputs only negative value irrespective of the class.<br>
…</p>
<pre><code class="lang-auto">    def forward(self, X):
        lstm_out, self.hidden = self.lstm(X, self.hidden)
        out = lstm_out.contiguous().view(1, -1)
        out = self.linear(out)
        return out
...
        if self.loss == 'BCE':
            self.criterion = net.BCEWithLogitsLoss().cuda()
</code></pre>
</blockquote>
</aside>
<p>First, I don’t see a <code>sigmoid()</code> anywhere in the code you posted.<br>
(I suppose that there could be one hiding in the superclass.)</p>
<p>Next, your model outputs the result of a <code>Linear</code> layer, which, in<br>
general, can produce negative values.</p>
<p>Last, your use of <code>BCEWithLogitsLoss</code> suggests that you are <em>not</em><br>
expecting positive probabilities produced by <code>sigmoid()</code>, but rather<br>
unbounded (including negative) <em>logits</em> (“raw scores”) produced by<br>
<code>Linear</code>.</p>
<p>If this doesn’t clear things up, please explain in more detail why you<br>
are expecting your outputs to be positive.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Atharva_Dubey: <p>Thanks a lot <a class="mention" href="/u/kfrank">@KFrank</a>.<br>
I had read that BCEWithLogitsLoss() essentialy has a built sigmoid in it, so during the training it was alright, but when predicting for an individual input, I will have to use a sigmoid on the output.  Is my understanding correct?</p>

KFrank: <p>Hello Atharva!</p>
<aside class="quote no-group" data-username="Atharva_Dubey" data-post="3" data-topic="67903" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/atharva_dubey/48/12127_2.png" class="avatar"> Atharva_Dubey:</div>
<blockquote>
<p>Thanks a lot <a class="mention" href="/u/kfrank">@KFrank</a>.<br>
I had read that BCEWithLogitsLoss() essentialy has a built sigmoid in it, so during the training it was alright, but when predicting for an individual input, I will have to use a sigmoid on the output.  Is my understanding correct?</p>
</blockquote>
</aside>
<p>It depends on what you mean by “predict.”</p>
<p><code>sigmoid()</code> converts logits to probabilities.  <code>sigmoid (0) = 1/2</code>.</p>
<p>If you wish to predict the value of a binary variable, that is, say<br>
whether your best guess for the value of the variable is 0 or 1,<br>
then it is sensible to say that if the probability of the variable<br>
being 1 is greater than 1/2, you will predict 1; otherwise you<br>
will predict 0.</p>
<p>So, <code>prediction = (probability &gt;= 1/2)</code>.  But this is the<br>
same as <code>prediction = (logit &gt;= 0)</code>.  So you don’t need<br>
the <code>sigmoid()</code> or the probability to make this kind of prediction.<br>
Simply take the output of your model (which is the output of a<br>
<code>Linear</code>, and should be understood as a logit) and compare it to 0.</p>
<p>But if by “predict” you mean that you want to state the probability<br>
of your variable being 1, then you do, of course, need to convert<br>
the logit to a probability by passing it through a <code>sigmoid()</code>.</p>
<p>When you compute the <em>accuracy</em> of your model, you usually<br>
compare 0-or-1-valued predictions with 0-or-1-value targets,<br>
so you don’t need <code>sigmoid()</code> – you just compare your logits<br>
with 0 to get the 0-or-1 (False-or-True) prediction.</p>
<p>When you compute the <em>loss</em> for your model, you often use<br>
cross-entropy, which works with probabilities, so the computation<br>
needs to convert the logits to probabilities.  You can do this<br>
yourself with an explicit <code>sigmoid()</code>, but for numerical reasons,<br>
it is better to use <code>BCEWithLogitsLoss</code> which has (in effect) a<br>
<code>sigmoid()</code> built into it.</p>
<p>Best.</p>
<p>K. Frank</p>