Why my loss function's value doesn't going down?

jaeyung1001: <p>my model is like below:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4909, 1500)
        self.relu1 = nn.ReLU()
        self.dout = nn.Dropout(0.2)
        self.fc2 = nn.Linear(1500, 300)
        self.prelu = nn.PReLU(1)
        self.out = nn.Linear(300, 1)
        self.out_act = nn.Sigmoid()
        
    def forward(self, input_):
        a1 = self.fc1(input_)
        h1 = self.relu1(a1)
        dout = self.dout(h1)
        a2 = self.fc2(dout)
        h2 = self.prelu(a2)
        a3 = self.out(h2)
        y = self.out_act(a3)
        return y
</code></pre>
<p>and… define my model and loss function &amp; optimizer function</p>
<pre><code class="lang-auto">model = Net()

criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=0.01, betas=(0.9, 0.999))
</code></pre>
<p>now start to train:</p>
<pre><code class="lang-auto">def train(epoch):
    model.train()
    for i in range(len(x_train_tensor)):
        optimizer.zero_grad()
        output = model(x_train_tensor[i])
        print(output.data[0], y_train_tensor[i])
        loss = criterion(output.data[0], y_train_tensor[i])
        loss.backward()
        optimizer.step()
        losses.append(loss.data[0])
        print("loss: {}".format(loss.data[0]))
</code></pre>
<p>but… output is like below</p>
<blockquote>
<p>tensor(0.4984) tensor(1., grad_fn=)<br>
loss: 0.47467532753944397<br>
tensor(0.5021) tensor(1., grad_fn=)<br>
loss: 0.4732956886291504<br>
tensor(0.5000) tensor(0., grad_fn=)<br>
loss: 0.9740557670593262<br>
tensor(0.4942) tensor(1., grad_fn=)</p>
</blockquote>
<p>when the label is 1 loss value is almost 0.4, but when a label is 0 loss function up to 0.9. these values repeat and don’t go to low value.</p>
<p>How can I resolve it? thank you!</p>

ptrblck: <p><code>nn.Sigmoid</code> and <code>nn.BCEWithLogitsLoss</code> don’t fit together.<br>
Either remove the <code>nn.Simgoid</code> or use <code>nn.BCELoss</code>.</p>

jaeyung1001: <p>if i change the nn.BCEWithLogitsLoss to nn.BCELoss function error was:</p>
<blockquote>
<p>RuntimeError: the derivative for ‘target’ is not implemented</p>
</blockquote>
<p>but remove nn.Sigmoid and leave BCEWithLogitsLoss there is no error occured</p>

ptrblck: <p>Oh wait, there seems to be another issue.<br>
Could you try to pass the <code>ouput</code> directly to your criterion instead of <code>output.data[0]</code>?</p>

jaeyung1001: <p>oh… same error… haha</p>

ptrblck: <p>I’m not sure why you get this error.<br>
Your code runs fine for this dummy input and target:</p>
<pre><code class="lang-python">def train(epoch):
    model.train()
    for i in range(len(x)):
        optimizer.zero_grad()
        output = model(x[i])
        print(output, target[i])
        loss = criterion(output, target[i])
        loss.backward()
        optimizer.step()
        losses.append(loss.item())
        print("loss: {}".format(loss.item()))

x = torch.randn(1, 4909)
target = torch.tensor([[1.]])

losses = []
train(0)
</code></pre>

Santhoshnumberone: <p>In theory the error during training is called <code>bias</code>.<br>
Ways to reduce bias</p>
<ol>
<li>
<p>Increase the number of hidden layers</p>
</li>
<li>
<p>change the NN architecture</p>
</li>
<li>
<p>Train NN for longer</p>
</li>
</ol>
<p>MAYBE! Threshold you are using for dropout might be too high, which are causing more units/neurons to turn off as you train longer.</p>

jaeyung1001: <p>I extracted my data from csv file, and define like below:</p>
<pre><code class="lang-auto">x_train_tensor = Variable(torch.FloatTensor(x_train.values))
x_test_tensor = Variable(torch.FloatTensor(x_test.values))
y_train_tensor = Variable(torch.Tensor(y_train.values), requires_grad = True)
y_test_tensor = Variable(torch.Tensor(y_test.values), requires_grad = True)
</code></pre>
<p>is there any problem ?</p>

ptrblck: <p>Could you try to use <code>torch.from_numpy()</code> to get the tensors instead of wrapping the numpy arrays directly? It’s the recommended way, but I’m not sure if it’s related to this issue.</p>

phdproblems: <p>My experience with this was different, when I was training a test model to test this exact thing I noticed that when I applied <code>nn.Sigmoid</code> on model output and used <code>BCELoss()</code> I got very bad results, my loss actually went to <code>NaN</code> after some iterations.</p>
<p>Similarly when I did not use <code>nn.Sigmoid</code> at model output and used <code>BCEWithLogitsLoss()</code> I agaain got bad results, no more NaN’s but error was not dropping from 0.999</p>
<p>Then I used <code>nn.Sigmoid</code> and <code>BCEWithLogitsLoss()</code> and got expected results, loss was dropping and so was error.</p>
<p>So can you please mention why these two dont go well together whereas my tests showed them working together.</p>
<p>Thanks</p>

ptrblck: <p>Since <code>sigmoid</code> will be applied twice in this (wrong) approach, you might have scaled down the gradients, thus stabilized the training, e.g. if your learning rate was too high.<br>
Here is a small example showing this effect:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 1)
)

data = torch.randn(1, 10)
target = torch.randint(0, 2, (1, 1)).float()

# 1) nn.BCEWithLogitsLoss
output = model(data)
loss = F.binary_cross_entropy_with_logits(output, target)
loss.backward()

print(model[0].weight.grad.norm())
&gt; tensor(0.1741)
print(model[2].weight.grad.norm())
&gt; tensor(0.2671)

# 2) nn.BCELoss
model.zero_grad()
output = model(data)
loss = F.binary_cross_entropy(torch.sigmoid(output), target)
loss.backward()
print(model[0].weight.grad.norm())
&gt; tensor(0.1741)
print(model[2].weight.grad.norm())
&gt; tensor(0.2671)

# 3) wrong
model.zero_grad()
output = model(data)
loss = F.binary_cross_entropy_with_logits(torch.sigmoid(output), target)
loss.backward()
print(model[0].weight.grad.norm())
&gt; tensor(0.0595)
print(model[2].weight.grad.norm())
&gt; tensor(0.0914)
</code></pre>
<p>Your loss might blow up and get eventually a <code>NaN</code> value, e.g. if the learning rate is set too high, which would also fit my assumption.</p>
<p>While applying <code>sigmoid</code> twice might have helped in your use case, I would recommend to try to debug the exploding loss (or <code>NaN</code> values).</p>

phdproblems: <p>Many thanks to giving such a helpful reply to such an old topic <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:"></p>
<p>This really helped me a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>