Output of regression model always 0 or 1

chenjesu: <p>I have tried both MSE and KLDiv losses, and everything I can think of / search for online.</p>
<p>The model always starts with a real number output and during training immediately gravitates toward 0 or 1, even though most of the target values are fractional numbers.</p>
<p>Here is my model (apologies for any typos, I am typing this out):</p>
<pre><code class="lang-auto">class Net(nn.Module):
   def __init__(self):
      super(Net, self).__init__()
      self.fc1 = nn.Linear(240*120*3, 300)
      self.fc2 = nn.Linear(300,3)
   def forward(self, x):
      x = torch.flatten(x, 1)
      x = self.fc1(x)
      x = F.relu(x)
      x = self.fc2(x)
      output = torch.sigmoid(x,)
      return output
</code></pre>
<p>Here is my training loop:</p>
<pre><code class="lang-auto">def train(model, device, train_loader, optimizer, lossfn, epoch):
   model.train(False)
   for b_idx, (data, target) in enumerate(train_loader):
      data, target = data.to(device), target.to(device)
      optimizer.zero_grad()
      output = model(data)
      loss = lossfn(output, target)
      print(output, target, loss)
      loss.backward()
      optimizer.step()
</code></pre>
<p>The loss function is both nn.KLDivLoss/MSELoss<br>
Both result in the same output for all inputs, and all the outputs are 0 or 1.</p>
<p>MSELoss:</p>
<pre><code class="lang-auto">output: tensor([[0., 1., 1.],
                       [0., 1., 1.],
                       [0., 1., 1.],
                       [0., 1., 1.],
                       [0., 1., 1.],
                       [0., 1., 1.],
                    ...]], device='cuda:0', grad_fn=&lt;SigmoidBackward&gt;)
target: tensor([[0.3438, 0.5781, 0.9688],
                       [0.0000, 0.6562, 0.0000],
                       [0.3438, 0.5781, 0.9688],
                       [0.3438, 0.5781, 0.9688],
                       [0.0000, 0.0000, 0.0000],
                       [0.3438, 0.5781, 0.9688],
                    ...]], device='cuda:0')                       
</code></pre>
<p>KLDiv loss is pretty much the same except all of the outputs are [1., 1., 1.] instead of [0., 1., 1.].</p>
<p>Before training, the outputs look much more normal:</p>
<pre><code class="lang-auto">output: tensor([[0.5651, 0.4700, 0.5182],
                        [0.5704, 0.4738, 0.5086],
                     ....etc]]
</code></pre>
<p>I would guess the issue is with how I’m performing the loss, since the outputs look fine before training.<br>
Any help would be appreciated.</p>

ptrblck: <p>I would guess this is the “nature” of the sigmoid activation, as the gradients will have the highest value at <code>f(x=0) = 0.5</code>, which would push the output towards the limits.</p>
<p>Usually you would use this activation for a binary or multi-label classification use case.<br>
If you are dealing with a regression, I would try to remove the output activation and check, if it’ll work.</p>

maze09: <p>Can this be solved by using tanh to restrict the output ?</p>

ptrblck: <p>I don’t know, if this was the issue, as the author of the post never followed up.<br>
Note that <code>tanh</code> would have a similar behavior of the gradient and of course your output would be in the range <code>[-1, 1]</code>, but you could try it out for your model.</p>

maze09: <p>Thank you ! I will give it a try !</p>

chenjesu: <p>This was the issue.  If I recall correctly, I discretized the output data and treated it like a multiclass classification problem to fix the issue, which wouldn’t work for all problems like this.  Using a linear activation for the output layer is a good thing to try if gradients are a problem.</p>