Loss always converges to around 1

Pepi: <p>I have no idea why this is happening…</p>
<p>I built this model a couple of days ago and it worked with good training and predictions.</p>
<p>However I opened the file today as I needed to add some things to the code (unrelated to the model) and now when I train it the loss stays at around 1 no matter how long I train it for - 10, 100, 1000, 10000 epochs.</p>
<p>I literally have not changed anything.</p>
<p>Anyway here is the relevant code:</p>
<pre><code class="lang-auto">class FundedDateNN(nn.Module):
    
    def __init__(self, input_size, hidden_size, output_size=1):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, output_size)
        self.init_weights()
    
    def init_weights(self):
        initrange = 0.5
        self.fc1.weight.data.uniform_(-initrange, initrange)
        self.fc1.bias.data.zero_()
        self.fc2.weight.data.uniform_(-initrange, initrange)
        self.fc2.bias.data.zero_()
        
    def forward(self, x):
        x = self.fc1(x)
        return self.fc2(x)
    
    def predict(self,x):
        return self.forward(x)
</code></pre>
<pre><code class="lang-auto">hidden_size = 20

model = FundedDateNN(input_size, hidden_size) 
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
</code></pre>
<pre><code class="lang-auto">def train(model, dataloader, num_epochs):

    model.train()
    losses = list()
    ts = time.time()
    for epoch in range(num_epochs):
        epoch_losses = list()
        for idx, (X, y) in enumerate(trainloader):

            optimizer.zero_grad()
            out = model(X)
            loss = criterion(out.squeeze(), y)
            epoch_losses.append(loss.item())

            loss.backward()
        optimizer.step()
        losses.append(np.mean(epoch_losses))
        print('Epoch: {}, loss: {}'.format(epoch, np.mean(epoch_losses)))
    te = time.time()
    fig, ax = plt.subplots()
    ax.plot(range(num_epochs), losses)
    plt.show()
    mins = int((te-ts) / 60)
    secs = int((te-ts) % 60)
    print('Training completed in {} minutes, {} seconds.'.format(mins, secs))
    return losses, model
</code></pre>
<pre><code class="lang-auto">n_epochs = 100
losses, model = train(model, trainloader, n_epochs)
</code></pre>
<p>I would really, really appreciate some help with this.</p>

ptrblck: <p>Your model can be seen as a single linear layer, since you are not using any activation function between <code>fc1</code> and <code>fc2</code> so you might want to add it.<br>
Also, make sure that <code>out.squeeze()</code> and <code>y</code> have the same shape as otherwise unwanted broadcasting could be applied and you should get a warning about this behavior.</p>

Pepi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks for your response!</p>
<p>Yes I understand that there is no activation between the two layers. I have tried a couple: Tanh, Relu and Sigmoid that all yield a similar output.</p>
<p>And yes, I have that call there to ensure they are the same shape.</p>
<p>Could it be a problem with the data itself?</p>

ptrblck: <aside class="quote no-group" data-username="Pepi" data-post="3" data-topic="96406">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pepi/48/28608_2.png" class="avatar"> Pepi:</div>
<blockquote>
<p>Could it be a problem with the data itself?</p>
</blockquote>
</aside>
<p>Could be the case, e.g. if the ranges are large and you don’t normalize the data.<br>
I assume that you don’t expect mismatches in the data, which could break the training.</p>
<p>As said before, your model is quite “small” (single linear layer), which might not be suitable to fit the data.</p>

Pepi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>After some further experimentation, I found that it doesn’t actually converge to 1. More specifically, it just learns very very slowly.</p>
<p>If when I start the training the loss is around 0.96. It drops only about 0.08 in 30,000 epochs.</p>
<p>I have been adjusting the learning rate as well as other hyperparameters. What steps can I take to improve how quickly the model learns?</p>
<p>I am Z scoring the data before training.</p>
<p>Yeah the model is quite small, I have since added a Tanh between tb two linear layers. How might I improve the model architecture?</p>
<p>I’m trying to solve a regression problem using a combination of continuous and categorical features.</p>
<p>Thank you so much for your help.</p>

ptrblck: <aside class="quote no-group" data-username="Pepi" data-post="5" data-topic="96406">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pepi/48/28608_2.png" class="avatar"> Pepi:</div>
<blockquote>
<p>How might I improve the model architecture?</p>
</blockquote>
</aside>
<p>You could search for similar use cases and check if some architectures were successfully used before.<br>
If you cannot find such a model, you could try to run experiments and check which change in the architecture is beneficial for the use case. There are also meta-learning approaches, which could try to find a suitable architecture, but I’m unsure if the current methods would suite your use case.<br>
Anyway, lectures such as <a href="https://www.fast.ai/">FastAI</a> provide also some best practices to create models.</p>

Pepi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
After doing some research and making some changes to the model.</p>
<p>I have updated my model architecture as follows. It’s task is now to perform binary classification.</p>
<pre><code class="lang-auto">class StatusNN(nn.Module):
    
    def __init__(self, input_size, hidden_size, num_classes=1):
        super().__init__()
        self.fc1 = nn.Linear(input_size, 512)
        self.fc2 = nn.Linear(512, hidden_size)
        self.fc3 = nn.Linear(hidden_size, num_classes)
        self.sigmoid = nn.Sigmoid()
        self.relu = nn.ReLU()
    def forward(self, x):
        x = self.fc1(x)      # [batch_size, 512] 
        x = self.relu(x)     # [batch_size, 512]
        x = self.fc2(x)      # [batch_size, 64]
        x = self.relu(x)     # [batch_size, 64]
        x = self.fc3(x)      # [batch_size, 1]
        x = self.sigmoid(x)  # [batch_size, 1]
        return x
    
    def predict(self,x):
        out = self.forward(x).detach().numpy()
        return np.round(out)
</code></pre>
<p>I initiate the model, loss_fn and optmizer:</p>
<pre><code class="lang-auto">USE_CUDA = False #torch.cuda.is_available()
device = 'cuda' if USE_CUDA else 'cpu'

input_size = dataset_x.shape[1]
hidden_size = 512
lr = 1e-3

model = StatusNN(input_size, hidden_size).to(device)
criterion = nn.BCELoss().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.6)
</code></pre>
<p>And this is my training loop:</p>
<pre><code class="lang-auto">def train(model, dataloader, num_epochs):

    model.train()
    losses = list()
    ts = time.time()
    for epoch in range(num_epochs):
        epoch_losses = list()
        for idx, (X, y) in enumerate(trainloader):
            
            X, y = X.to(device), y.to(device)

            out = model(X)
            loss = criterion(out, y.unsqueeze(1))
            
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            epoch_losses.append(loss.item())
            
        losses.append(np.mean(epoch_losses))
        if epoch % 2 == 0:
            print('Epoch: {}, Loss: {}'.format(epoch, np.mean(epoch_losses)))
    te = time.time()
    
    fig, ax = plt.subplots()
    ax.plot(range(num_epochs), losses)
    plt.show()
    
    mins = int((te-ts) / 60)
    secs = int((te-ts) % 60)
    print('Training completed in {} minutes, {} seconds.'.format(mins, secs))
    
    return losses, model
</code></pre>
<p>However, even after all of this I still get this asymptotic, plateauing effect with my loss as it is not converging to 0, but some arbitrary value.</p>
<p>Could the issue lie in my training loop?</p>
<p>I have tried overfitting the model on the first batch to the same result. I have also tried balancing my dataset using SMOTE.</p>
<p>Thank you so much for your help so far, I really appreciate it.</p>

ptrblck: <p>Your model seems to work for random data, as I’m able to perfectly overfit a random input and target:</p>
<pre><code class="lang-python">class StatusNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes=1):
        super().__init__()
        self.fc1 = nn.Linear(input_size, 512)
        self.fc2 = nn.Linear(512, hidden_size)
        self.fc3 = nn.Linear(hidden_size, num_classes)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.fc1(x)      # [batch_size, 512] 
        x = self.relu(x)     # [batch_size, 512]
        x = self.fc2(x)      # [batch_size, 64]
        x = self.relu(x)     # [batch_size, 64]
        x = self.fc3(x)      # [batch_size, 1]
        return x

model = StatusNN(10, 512)
data = torch.randn(64, 10)
target = torch.randint(0, 2, (64, 1)).float()
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(100):
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print('Epoch {}, loss {}'.format(epoch, loss.item()))

model.eval()
output = model(data)
pred = torch.sigmoid(output) &gt; 0.5
print((pred == target).float().sum() / target.size(0))
&gt; tensor(1.)
</code></pre>
<p>Note that I replaced sigmoid + <code>nn.BCELoss</code> with logits + <code>nn.BCEWithLogitsLoss</code> for more numerical stability.</p>