Costant loss on simple MLP

username: ireneferfo
cooked: <p>Hello, I have the following net to perform a binary classification of some trajectories:</p>
<h4><a name="architecture-1" class="anchor" href="#architecture-1"></a>Architecture</h4>
<pre data-code-wrap="python"><code class="lang-python">class DCR(nn.Module):
    def __init__(self, kemb_size, nvar, points, device):
        super().__init__()
        self.phis = load_phis_dataset()
        self.kemb = get_kernel_embedding(self.phis, nvar, samples = kemb_size).to(device) # (concepts, kemb_size)
        _ = self.kemb.requires_grad_()
        
        self.fc1 = nn.Linear(kemb_size + (nvar*points), 64)
        self.fc2 = nn.Linear(64, 1)
        self.sigmoid = nn.Sigmoid()
                
    def forward(self, x):
        # concept truth degrees
        rhos = get_robustness(x, self.phis, time = False) # (trajectories, concepts)
        _ = rhos.requires_grad_()
        # embed trajectories in kernel space
        traj_emb = torch.matmul(rhos, self.kemb) # (trajectories, kemb_size)
        _ = traj_emb.requires_grad_()
        # combine info from traj_embed and x to predict class 
        x_new = x.view(x.size(0), -1) # flatten x
        combined_features = torch.cat((traj_emb, x_new), dim=1) # (trajectories, kemb_size + x.shape[0]*x.shape[1])

        output = self.fc1(combined_features)
        output = F.relu(output)
        output = self.fc2(output)
        output = self.sigmoid(output)
        
        return output.squeeze(1)
</code></pre>
<h4><a name="training-2" class="anchor" href="#training-2"></a>Training</h4>
<pre data-code-wrap="python"><code class="lang-python">model = DCR(kemb_size, nvar, points, device).to(device)
criterion = nn.BCELoss().to(device)
optimizer = optim.SGD(model.parameters(), lr=0.01)

model.train()
for epoch in range(10):
    epoch_loss = 0.0

    for batch, labels in train_loader:
        batch, labels = batch.to(device), labels.to(device)
                 
        y_preds = model(batch)
        loss = criterion(y_preds, labels.float())
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        epoch_loss += y_preds.shape[0] * loss.item()
  
    print(f'Epoch: {epoch}, Loss: {epoch_loss/len(y_train):.5f}')
</code></pre>
<p>However, the training loss remains perfectly constant at every epoch, and the weights are not updated. What could I be doing wrong?</p>
<p>I tried solving the problem with some <code>.requires_grad_()</code>, but it didn’t work.</p>
<h4><a name="some-explanations-3" class="anchor" href="#some-explanations-3"></a>Some explanations</h4>
<ul>
<li>phis = list of STL formulae</li>
<li>kemb = kernel embedding of said STL formulae</li>
<li>rhos = robustness of STL formulae on input trajectories</li>
</ul>
<p>All shapes are as comments in the code.</p>

username: ptrblck
cooked: <p>Remove the sigmoid and use <code>nn.BCEWithLogitsLoss</code> instead for a better numerical stability.</p>

username: stijnvdl
cooked: <p>On a related note: if using <code>BCEWithLogitsLoss</code> for numerical stability, what would be the recommended way to still get the output of the sigmoid during inference? Just passing the output of the network through a <code>torch.Tensor.sigmoid</code>?</p>

username: J_Johnson
cooked: <p>In the forward pass, you could just call:</p>
<pre><code class="lang-auto">if self.training:
    return x
else:
    return self.sigm(x)
</code></pre>

username: J_Johnson
cooked: <p><a class="mention" href="/u/ireneferfo">@ireneferfo</a><br>
There are several functions used in the forward pass that are undefined. It’s possible the break in logic happens in one of those. Can you share those here?</p>
<p>Also, have you tried seeing if the model would overfit to a small batch of randomized data and labels?</p>
<p>I.e.</p>
<pre><code class="lang-auto">batch = torch.rand(10, 100)
labels = torch.round(torch.rand(10)

for epoch in range(10):
    epoch_loss = 0.0
    for i in range(10000):
        #insert forward/backward pass in here

</code></pre>

username: stijnvdl
cooked: <p>Will this not go wrong when calculating the test/validation loss during training? Because it uses the same <code>BCEWithLogitsLoss</code> but the network would output the sigmoid, since you would call <code>model.eval()</code> to set <code>self.training=False</code> before evaluating the test/validation loss.</p>

username: J_Johnson
cooked: <p>I see. I guess it depends on how you’re evaluating your model. Otherwise, you could just call <code>F.sigmoid(x, dim = 1)</code> on your outputs when you need it.</p>

username: ireneferfo
cooked: <p>To share those functions I would need to share the whole repository of my research, and it’s way too big (and currently private). It is possibile that the logic breaks there, but I thought that if after the computations I’d call <code>.requires_grad_()</code> everything would be fine, being the outputs of those functions simple tensors of floats.</p>
<p>With a small batch of random data, without the sigmoid and with <code>nn.BCEWithLogitsLoss</code>, as suggested by <a class="mention" href="/u/ptrblck">@ptrblck</a>, the model does train. I guess the problem really is in the ‘hidden’ code, which I didn’t write so it’ll be a mess to fix.</p>
<p>Thanks for the help!</p>

username: J_Johnson
cooked: <p>Yes. It should still overfit a small set even with nn.BCELoss and a final sigmoid activation.</p>
<p>However, the nn.BCEWithLogitsLoss is a better habit to get into because it handles logits outside of the +10/-10 range better.</p>
<p>With the other preprocessing functions, you could try plotting out what those are giving for outputs to see if they are doing what you expect.</p>