Training Loss is Not Going Down

username: B_craig
cooked: <p>Hello, I am a begginer AI developer attempting to train my first model. I have gone through everything and can not figure out why my training loss will not go down. Going through it, I have found that it is because my gradient for the convolutional blocks are staying at 0.0 and I can not figure out why. Any help would be much appreciated</p>
<p>import torch<br>
from torch import nn<br>
from pathlib import Path<br>
from torch.utils.data import DataLoader<br>
from torchvision import datasets, transforms<br>
import os</p>
<p>device = “cuda” if torch.cuda.is_available() else “cpu”</p>
<p>DATA_PATH = Path(“data/”)<br>
imagePathList = list(DATA_PATH.glob(“<em>/</em>/*”)) # Creates list of paths to ALL images</p>
<p>data_transform = transforms.Compose([<br>
transforms.Resize(size=(64, 64)),<br>
transforms.RandomHorizontalFlip(p=0.5),<br>
transforms.ToTensor()<br>
])</p>
<p>trainData = datasets.ImageFolder(root=DATA_PATH/“train”, transform=data_transform)<br>
testData = datasets.ImageFolder(root=DATA_PATH/“test”, transform=data_transform)</p>
<p>classNames = trainData.classes</p>
<p>BATCH_SIZE = 1  # Adjusted for better training dynamics<br>
trainDataLoader = DataLoader(dataset=trainData, batch_size=BATCH_SIZE, shuffle=True)<br>
testDataLoader = DataLoader(dataset=testData, batch_size=BATCH_SIZE, shuffle=False)</p>
<p>class ASLModel(nn.Module):<br>
def <strong>init</strong>(self, input_shape, hidden_units, output_shapes):<br>
super().<strong>init</strong>()<br>
self.convolutional_block1 = nn.Sequential(<br>
nn.Conv2d(in_channels=input_shape, out_channels=hidden_units, kernel_size=3, stride=1, padding=1),<br>
nn.ReLU(),<br>
nn.Conv2d(in_channels=hidden_units, out_channels=hidden_units, kernel_size=3, stride=1, padding=1),<br>
nn.ReLU(),<br>
nn.MaxPool2d(kernel_size=2)<br>
)</p>
<pre><code>    self.convolutional_block2 = nn.Sequential(
        nn.Conv2d(in_channels=hidden_units, out_channels=hidden_units, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.Conv2d(in_channels=hidden_units, out_channels=hidden_units, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2)
    )

    self.classifier = nn.Sequential(
        nn.Flatten(),
        nn.Linear(in_features=hidden_units*32*32, # Adjusted for output size
                  out_features=output_shapes)
    )

def forward(self, x):
    x = self.convolutional_block1(x)
    x = self.convolutional_block2(x)
    x = self.classifier(x)
    return x
</code></pre>
<p>def accuracy_fn(y_true, y_pred):<br>
correct = torch.eq(y_true, y_pred).sum().item()<br>
acc = (correct / len(y_pred)) * 100<br>
return acc</p>
<p>model = ASLModel(input_shape=3, hidden_units=30, output_shapes=len(classNames))</p>
<p>lossFunction = nn.CrossEntropyLoss()<br>
modelOptimizer = torch.optim.SGD(params=model.parameters(), lr=0.1)  # Adjusted learning rate</p>
<p>def trainingModel(modelParam, dataLoader, optimizer, lossFN, accuracyFN, device=device):<br>
modelParam.train()<br>
trainingLoss = 0<br>
trainingAcc = 0<br>
modelParam.to(device)<br>
for batch, (X, y) in enumerate(dataLoader):<br>
X, y = X.to(device), y.to(device)<br>
y_pred = modelParam(X)<br>
loss = lossFN(y_pred, y)<br>
trainingLoss += loss<br>
<span class="hashtag-raw">#print</span>(f"loss={loss}, trainingLoss={trainingLoss}“)<br>
trainingAcc += accuracyFN(y_true=y, y_pred=y_pred.argmax(dim=1))<br>
optimizer.zero_grad()<br>
loss.backward()<br>
for name, param in model.named_parameters():<br>
if param.grad is not None:<br>
print(f"Gradient for {name}: {param.grad.mean()}”)<br>
else:<br>
print(f"Gradient for {name} is None")<br>
optimizer.step()</p>
<pre><code>trainingLoss /= len(dataLoader)
trainingAcc /= len(dataLoader)
print(f"Training Loss: {trainingLoss:.5f}, Training acc: {trainingAcc:.2f}")
</code></pre>
<p>def testingModel(modelParam, dataLoader, lossFN, accFN, device=device):<br>
modelParam.eval()<br>
testLoss, testAcc = 0, 0<br>
modelParam.to(device)<br>
with torch.inference_mode():<br>
for X, y in dataLoader:<br>
X, y = X.to(device), y.to(device)<br>
test_pred = modelParam(X)<br>
testLoss += lossFN(test_pred, y).item()<br>
testAcc += accFN(y_true=y, y_pred=test_pred.argmax(dim=1))</p>
<pre><code>testLoss /= len(dataLoader)
testAcc /= len(dataLoader)
print(f"Testing Loss: {testLoss:.5f}, Testing acc: {testAcc:.2f}")
</code></pre>
<p>epochs = 1<br>
for epoch in range(epochs):<br>
print(f"Epoch {epoch+1}")<br>
trainingModel(model, trainDataLoader, modelOptimizer, lossFunction, accuracy_fn, device)<br>
testingModel(model, testDataLoader, lossFunction, accuracy_fn, device)</p>

username: Soumya_Kundu
cooked: <p>Hi!</p>
<p>Some context: Machine Learning/Deep Learning is just not model dependent. When asking such questions as to why the loss is not working or etc., it is always helpful to provide context as to what your data is as well.</p>
<p>Additionally, please use the option to paste the code properly and possibly add a colab notebook to be able to reproduce your work (with your data if shareable).</p>
<p>Specifically for the quetion from what I can gauge:</p>
<ol>
<li>You could increase your batch size and learning rate.</li>
<li>add batchnorm</li>
</ol>

username: jrog
cooked: <p>Hi!</p>
<p>I was able to use your code with tmp data (cifar10), and I found that probably the main issue was ReLU combined with a too big learning rate. An excessively high learning rate combined with ReLU may push the activations into saturation regions (in case of ReLU, zeroing output of layers), zeroing the gradients.</p>
<p>Switching all activations to LeakyReLU (which prevents creating “dead neurons”), and lowering the learning rate to for example 0.001 (0.1 learning rate might be too high, and create problem of exploding gradients) solved the issue for me. As previously mentioned, using higher batch size woulde be beneficial as well</p>

username: mycul
cooked: <p>^This guy is correct!</p>
<p>Since you are a learner, I highly suggest checking out ‘vanishing gradients’ and ‘exploding gradients’.</p>
<p>It’ll begin to teach you more about how to stabilize your training loop, but also how computationally efficient the approach to stabilizing your training loop will be. Lots of trade-offs to be made. good luck!</p>