Trying to understand what's going on here

slothman3878: <p>I’m new to machine learning and pytorch and any insight would be greatly appreciated. I’ve been playing around with the MNIST number recognition dataset and thought it would be interesting to train different models on the negative images.</p>
<p>One model in particular yielded interesting results. The model is just three fully connected layers with the ReLU activation. This is what the learning process looks like.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e45c272c52db36e4a3ef6f0682235a6ecdaf81ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/e45c272c52db36e4a3ef6f0682235a6ecdaf81ec" title="network2-negative-01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45c272c52db36e4a3ef6f0682235a6ecdaf81ec_2_690x327.png" alt="network2-negative-01" data-base62-sha1="wAaiBXqyvSAjn4rLQXSfUGhZihu" width="690" height="327" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45c272c52db36e4a3ef6f0682235a6ecdaf81ec_2_690x327.png, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e45c272c52db36e4a3ef6f0682235a6ecdaf81ec.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e45c272c52db36e4a3ef6f0682235a6ecdaf81ec.png 2x" data-dominant-color="FAF9F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">network2-negative-01</span><span class="informations">989×470 44.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>At first I wondered if this was an exploding gradient issue due to high activation values, but when testing with a model with the sigmoid activation function resulted more consistent results and smoother changes in loss, and from my understanding models with sigmoid activations are more susceptible to gradient explosion.</p>
<p>Is it just some bug in my code? Here’s the code snippet just in case.</p>
<pre><code class="lang-auto">class Network2(nn.Module):
  def __init__(self):
    super(Network2, self).__init__()
    self.fc1 = nn.Linear(784, 200)
    self.fc2 = nn.Linear(200, 100)
    self.fc3 = nn.Linear(100, 10)
    
  def forward(self, x):
    x = torch.flatten(x, 1)
    x = self.fc1(x)
    x = nn.functional.relu(x)
    x = self.fc2(x)
    x = nn.functional.relu(x)
    x = self.fc3(x)
    x = nn.functional.relu(x)
    return x
</code></pre>
<pre><code class="lang-auto"># traning code
eta = 0.1
epochs = 20

net2 = network.Network2()
print(net2)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net2.parameters(), lr=eta)
net2.zero_grad()

val_accs = []
val_loss = []
train_accs = []
train_loss = []
grads = []
    
total_loss = 0.
total_acc = (0, 0)

for epoch in range(epochs):
    print(f"epoch {epoch}")
    
    for i, (X, y) in enumerate(train_loader):
        net2.train() # training mode
        
        output = net2(X)
        loss = criterion(output, y)
        total_loss += loss.item()
        total_acc = tuple(a + b for a, b in zip(total_acc, accuracy(output, y)))
        
        loss.backward()
        
        if i + 1 == len(train_loader):
            grads.append(net2.fc1.weight.grad.detach().numpy().flatten())
        
        optimizer.step()
        optimizer.zero_grad()

        if i % 100 == 99: # for every 100th batch
            net2.eval() # evaluation mode
            
            with torch.no_grad():
                total_val_loss = 0.
                val_acc = (0, 0)
                for j, (val_X, val_y) in enumerate(validation_loader):
                    # evaluation loss and accuracy check
                    output = net2(val_X)
                    val_acc = tuple(a + b for a, b in zip(val_acc, accuracy(output, val_y)))
                    total_val_loss += criterion(output, val_y).item()
                print(f" evaluation accuracy: {100 * val_acc[0] / val_acc[1]}")
                val_accs.append(val_acc)
                val_loss.append(total_val_loss / len(validation_loader))
                
            print(f" training accuracy: {100 * total_acc[0] / total_acc[1]}")
            train_accs.append(total_acc)
            train_loss.append(total_loss / 100)
                       
            total_loss = 0.
            total_acc = (0, 0)
</code></pre>

slothman3878: <p>Here are a few more tests with the same model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6eca8700da492b73007123a455b60ccd0c1cd155.png" data-download-href="https://discuss.pytorch.org/uploads/default/6eca8700da492b73007123a455b60ccd0c1cd155" title="network2-negative-02"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6eca8700da492b73007123a455b60ccd0c1cd155_2_690x327.png" alt="network2-negative-02" data-base62-sha1="fO6qWDKrqMpFpub1i28BbcHIBnv" width="690" height="327" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6eca8700da492b73007123a455b60ccd0c1cd155_2_690x327.png, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6eca8700da492b73007123a455b60ccd0c1cd155.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6eca8700da492b73007123a455b60ccd0c1cd155.png 2x" data-dominant-color="FAF9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">network2-negative-02</span><span class="informations">989×470 42.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

slothman3878: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e48ee9cfc600eb89a3e23f3aa11a314d1dddcbe.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e48ee9cfc600eb89a3e23f3aa11a314d1dddcbe" title="network2-negative-03"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e48ee9cfc600eb89a3e23f3aa11a314d1dddcbe_2_690x327.png" alt="network2-negative-03" data-base62-sha1="fJCMeH5D1BkVb8L7IDRNYpL1uoK" width="690" height="327" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e48ee9cfc600eb89a3e23f3aa11a314d1dddcbe_2_690x327.png, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e48ee9cfc600eb89a3e23f3aa11a314d1dddcbe.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e48ee9cfc600eb89a3e23f3aa11a314d1dddcbe.png 2x" data-dominant-color="FAF9F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">network2-negative-03</span><span class="informations">989×470 42.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>