How to interpret the output of a single neuron?

alx: <p>Hi, I’m currently using CrossEntropyLoss for a model with a single neuron as ouptut.</p>
<p>My predictions have the following pattern:</p>
<pre><code class="lang-auto">        [ 0.0445],
        [-0.1698],
        [-0.1343],
        [-0.5633],
        [ 0.2206],
</code></pre>
<p>To get my classes, do I have to do:</p>
<pre><code class="lang-auto">      pred_batch = torch.sigmoid(pred_batch)
      pred_batch[pred_batch &gt;= 0.5] = 1
      pred_batch[pred_batch &lt; 0.5] = 0
</code></pre>
<p>or</p>
<pre><code class="lang-auto">      pred_batch[pred_batch &gt;= 0] = 1
      pred_batch[pred_batch &lt; 0] = 0
</code></pre>
<p>Also, I’m getting some negative loss for my training set, is that normal?</p>

chetan_patil: <p>Firstly, <code>nn.CrossEntropyLoss()</code> is mostly used when you need multple predictions.<br>
Since you have just one output neuron, you must use <code>nn.Sigmoid()</code> followed by <code>nn.BCELoss()</code> or simply <code>nn.BCELossWithLogits()</code>.</p>
<p>Just need a clarification,</p>
<ol>
<li>You have single output neuron.</li>
<li>Your target spans between 0 and 1.</li>
</ol>
<p>Consider a batch-size of 7.</p>
<ol>
<li>pred_batch will be of shape (7,1) since u have single neuron.</li>
<li>target shape will be (7,) with minimum value of 0 and maximum value of 1.<br>
Right ?</li>
</ol>
<p>How come you are able to pass these inside <code>CrossEntropyLoss()</code> without getting any error.?</p>

alx: <p>Good question, I have no idea.</p>
<p>When I use <code>BCELossWithLogits()</code> I had to do a<br>
<code>label_batch = label_batch.view(-1, 1)</code></p>

chetan_patil: <p>Didn’t <code>BCELosswithLogits()</code> do the purpose then ?<br>
If you were using this bce-with-logits, then <code>torch.sigmoid()</code> is the right way as you have posted in the question.<br>
If you’ve used <code>nn.CrossEntropyLoss()</code>, then u apply Softmax(), and pick the class which has the highest value.<br>
In your case, picking the class with highest value would always result in the same class ( zeroth class ), since you have just one neuron.</p>

alx: <p>Here’s my code:</p>
<pre><code class="lang-auto">    model = models.resnet50(pretrained=pretrain_status)
    num_ftrs = model.fc.in_features
    model.fc = nn.Sequential(
      nn.Dropout(dropout_rate),
      nn.Linear(num_ftrs, 1))
</code></pre>
<pre><code class="lang-auto">  criterion = nn.BCELossWithLogits() 
  optimizer = torch.optim.SGD(model.parameters(), lr =lr)
</code></pre>
<pre><code class="lang-auto">  model.train()
  for image_batch, label_batch in tqdm(train_dl):
    
    optimizer.zero_grad()
    image_batch, label_batch = image_batch.to(DEVICE), label_batch.to(DEVICE)
    pred_batch = model(image_batch)
   
    label_batch = label_batch.view(-1, 1)

    batch_loss = criterion(pred_batch.double(), label_batch.double())
    total_train_loss += batch_loss.item()

    batch_loss.backward()
    optimizer.step()

    pred_batch[pred_batch &gt;= 0] = 1
    pred_batch[pred_batch &lt; 0] = 0
</code></pre>
<p>I know there’s a mistake in there but I can’t seem to find it</p>

chetan_patil: <p>Hi <a class="mention" href="/u/alx">@alx</a>,<br>
the code is perfectly fine upto <code>optimizer.step()</code></p>
<p>Please add these steps after that line</p>
<pre><code class="lang-auto">probabilities  = torch.sigmoid(pred_batch)
pred_batch[probabilities&gt;=0.5] = 1
pred_batch[probabilities&lt;0.5]  = 0
</code></pre>
<p>This is the first case which you have asked in your question.!</p>