Issues with training a model

username: milan_kalkenings
cooked: <p>Hi all,<br>
I am new to pytorch and I wonder why my neural network doesn’t train. The loss is always the same in every epoch and the execution time per epoch implicates that no training really happens.</p>
<p>Sry for the long post…</p>
<p>class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(11, 6),<br>
nn.ReLU(),<br>
nn.Linear(6, 4),<br>
nn.ReLU(),<br>
nn.Linear(4, 1),<br>
nn.Softmax()<br>
)</p>
<pre><code>def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits
</code></pre>
<p>model = NeuralNetwork().to(‘cuda’)</p>
<p>for n, p in model.named_parameters():<br>
print(p.device, “”, n)</p>
<p><span class="hashtag">#hyperparameters</span> and optimization<br>
learning_rate = 0.001<br>
batch_size = 128<br>
epochs = 5<br>
loss_fn = nn.BCELoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</p>
<p>training_data = CustomDataset(data_dir=“train.csv”, targets=“quality”)<br>
test_data = CustomDataset(data_dir=“test.csv”, targets=“quality”)</p>
<p>train_dataloader = DataLoader(training_data, batch_size=batch_size)<br>
test_dataloader = DataLoader(test_data, batch_size=batch_size)</p>
<p>def train_loop(dataloader, model, loss_fn, optimizer):<br>
for batch, (X, y) in enumerate(dataloader):<br>
# move the data to the gpu<br>
X = X.to(‘cuda’)<br>
y = y.to(‘cuda’)</p>
<pre><code>    y = y.float() 
    
    # Compute prediction and loss
    pred = model(X.float())
    
    y = torch.reshape(input=y, shape=pred.shape)
    loss = loss_fn(pred, y)

    # Backpropagation
    optimizer.zero_grad() 
    loss.backward()
    optimizer.step()
</code></pre>
<p>def test_loop(dataloader, model, loss_fn):<br>
size = len(dataloader.dataset)<br>
print(size)<br>
test_loss = 0</p>
<pre><code>with torch.no_grad():
    for X, y in dataloader:
        X = X.float().to('cuda')
        y = y.to('cuda')
        y = y.float()
        pred = model(X)
        y = torch.reshape(input=y, shape=pred.shape)
        test_loss += loss_fn(pred, y).item()
        
test_loss /= size
print(f"Avg loss on test: {test_loss} ")
</code></pre>
<p>for t in range(epochs):<br>
print(f"Epoch {t+1}\n-------------------------------")<br>
start = timeit.default_timer()<br>
train_loop(train_dataloader, model, loss_fn, optimizer)<br>
test_loop(test_dataloader, model, loss_fn)<br>
stop = timeit.default_timer()<br>
print("time: ", stop - start, “\n”)<br>
print(“Done!”)</p>
<h2>output:<br>
Epoch 1</h2>
<p>320<br>
Avg loss on test: 0.4052734375<br>
time:  0.7036787000000002</p>
<h2>Epoch 2</h2>
<p>320<br>
Avg loss on test: 0.4052734375<br>
time:  0.023357099999999242</p>
<h2>Epoch 3</h2>
<p>320<br>
Avg loss on test: 0.4052734375<br>
time:  0.02345839999999999</p>
<h2>Epoch 4</h2>
<p>320<br>
Avg loss on test: 0.4052734375<br>
time:  0.0254683</p>
<h2>Epoch 5</h2>
<p>320<br>
Avg loss on test: 0.4052734375<br>
time:  0.02313960000000037</p>
<p>Done!</p>

username: ptrblck
cooked: <p>I assume you are working on a binary classification.<br>
If that’s the case, note that <code>nn.Softmax</code> would output a tensor of ones for an input tensor (activation coming from the previous linear layer) in the shape <code>[batch_size, 1]</code>, so you might want the use <code>nn.Sigmoid</code> instead.<br>
For better numerical stability I would recommend to remove the activation (no softmax or sigmoid) and use <code>nn.BCEWithLogitsLoss</code> as the criterion.</p>