Multi-label classification training loss not decreasing

chinmaya_mahesh: <p>Hi,<br>
As a simple test for multi-label classification I am trying to fit a simple linear neural network to random noise and randomly generated binary labels for a few categories. However, over each epoch the loss seems to not decrease at all. Here is a self-contained example:</p>
<pre><code class="lang-auto">import os
import torch
import torch.onnx as onnx
import torchvision.models as models
import numpy as np
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets
from torchvision.transforms import ToTensor, Lambda
import torch.nn.functional as F
from torch.autograd import Variable

X_np = np.random.rand(1000,214)
y_np = np.random.randint(2, size=(1000,15))

train_X = X_np[:900]
train_y = y_np[:900]
test_X = X_np[900:]
test_y = y_np[900:]


train_X_tensor = Variable(torch.from_numpy(train_X).float(), requires_grad=True)
train_y_tensor = Variable(torch.from_numpy(train_y).float(), requires_grad=False)
test_X_tensor = Variable(torch.from_numpy(test_X).float(), requires_grad=True)
test_y_tensor = Variable(torch.from_numpy(test_y).float(), requires_grad=False)


train_dataset = TensorDataset(train_X_tensor, train_y_tensor)
train_dataloader = DataLoader(train_dataset,batch_size=1)
test_dataset = TensorDataset(test_X_tensor, test_y_tensor)
test_dataloader = DataLoader(test_dataset,batch_size=1)

device='cpu'
learning_rate = 0.3
epochs = 5
num_triggers=len(train_y[0])
num_inputs=len(train_X[0])
lambda1 = 0.001

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()

        self.linear1 = nn.Linear(num_inputs, num_triggers)

    def forward(self, x):
        x = self.linear1(x)
        x2 = torch.sigmoid(x)
        return x2

model = NeuralNetwork().to(device)

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for i in range(3):
        model.train()
        print("epoch", i+1)
        for batch, (X, y) in enumerate(dataloader):
            X, y = X.to(device), y.to(device)
            # Compute prediction and loss
            pred = model(X)
            loss = loss_fn(pred, y)

            # Backpropagation
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if batch % 100 == 0:
                loss, current = loss.item(), batch * len(X)
                print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")
        test_loop(test_dataloader, model, loss_fn)


def test_loop(dataloader, model, loss_fn):
    model.eval()
    size = len(dataloader.dataset)
    test_loss, correct = 0, 0

    with torch.no_grad():
        for X, y in dataloader:
            pred = model(X)
            test_loss += loss_fn(pred, y).item()

    test_loss /= size
    print(f"Test Error: \n Avg loss: {test_loss:&gt;8f} \n")
    return [],[]


loss_fn = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

train_loop(train_dataloader, model, loss_fn, optimizer)
</code></pre>
<p>I feel like I might be misunderstanding something in either how I’m using the randomly generated numpy array or how I’m training the neural network. Any advice would be appreciated, thanks!</p>
<p>PS: It is pretty important that I read in the data from a numpy array as that is how I will receive my training data in my use-case.</p>

ptrblck: <p>I wouldn’t expect the validation loss to decrease, as you are trying to overfit randomly generated samples, so the model won’t be able to “generalize” somehow to random validation samples.<br>
However, you might be able to overfit the training dataset.<br>
To do so, use a tiny dataset first (e.g. just 10 samples), as your model is only a single linear layer.<br>
Also, remove the <code>sigmoid</code> and use <code>nn.BCEWithLogitsLoss</code> instead. Play around with some other hyperparameters, such as the optimizer (you might try to change it to e.g. <code>Adam</code> and lower the learning rate to e.g. <code>1e-3</code>).<br>
<code>Variable</code>s are also deprecated, so remove them, too.</p>