Loss doesn't decrease

Preetham_R_Patlolla: <p>I’ve been trying to develop an Autoencoder model for the task of knowledge representation where my input is a sequence of images. Loss of this model (both training and testing) doesn’t decrease. It is instead fluctuating and of course, my image reconstruction is very poor. I’ve tried the following:</p>
<ol>
<li>Changing the learning rate between 1 and 0.001</li>
<li>Increasing and decreasing the batch size</li>
<li>With and without dropout layers</li>
</ol>
<h1>Encoder-Decoder model</h1>
<p>class Encoder_Decoder(nn.Module):</p>
<pre><code>def __init__(self):
    super(Encoder_Decoder, self).__init__()
    
    #Encoder
    self.encoder = nn.Sequential(nn.Conv3d(in_channels=3, out_channels=16, kernel_size=(3, 3, 3), padding=(0,0,0)),  
                                 nn.ReLU(), nn.BatchNorm3d(16), 
                                 nn.MaxPool3d(kernel_size=(1,2,2)),
                                 nn.Conv3d(in_channels=16, out_channels=64, kernel_size=(3, 3, 3), padding=(0,1,1)), 
                                 nn.ReLU(), nn.BatchNorm3d(64), 
                                 nn.MaxPool3d(kernel_size=(2,2,2)),
                                 nn.Conv3d(in_channels=64, out_channels=256, kernel_size=(3, 3, 3), padding=(1,1,1)), 
                                 nn.ReLU(), nn.BatchNorm3d(256), 
                                 nn.MaxPool3d(kernel_size=(2,2,2)),
                                 nn.Conv3d(in_channels=256, out_channels=64, kernel_size=(3,3,3),padding=(1,1,1)), 
                                 nn.ReLU(), nn.BatchNorm3d(64), 
                                 nn.MaxPool3d(kernel_size=(2,2,2)),
                                 nn.Conv3d(in_channels=64, out_channels=16, kernel_size=(1,1,1), padding=(0,0,0)), 
                                 nn.ReLU(), nn.BatchNorm3d(16), 
                                 nn.MaxPool3d(kernel_size=(2,1,1)),
                                 nn.Conv3d(in_channels=16, out_channels=4, kernel_size=(1,1,1), padding=(0,0,0)), 
                                 nn.ReLU(), nn.BatchNorm3d(4), 
                                 nn.MaxPool3d(kernel_size=(1,1,1)))
    
    
    #Decoder
    self.decoder = nn.Sequential(nn.ConvTranspose3d(in_channels=4, out_channels=16, kernel_size=3), 
                                 nn.ReLU(), 
                                 nn.Upsample(scale_factor=(2,2,2)),
                                 nn.ConvTranspose3d(in_channels=16, out_channels=64, kernel_size=3), 
                                 nn.ReLU(), 
                                 nn.Upsample(scale_factor=(2,2,2)),
                                 nn.ConvTranspose3d(in_channels=64, out_channels=256, kernel_size=3), 
                                 nn.ReLU(), 
                                 nn.Upsample(scale_factor=(2,2,2)),
                                 nn.ConvTranspose3d(in_channels=256, out_channels=64, kernel_size=3), 
                                 nn.ReLU(), 
                                 nn.Upsample(scale_factor=(2,2,2)),
                                 nn.ConvTranspose3d(in_channels=64, out_channels=16, kernel_size=1), 
                                 nn.ReLU(), 
                                 nn.Upsample(size=(22,28,28)),
                                 nn.ConvTranspose3d(in_channels=16, out_channels=3, kernel_size=1), 
                                 nn.ReLU(), 
                                 nn.Upsample(size=(22,28,28)))
    
def forward(self,x):

    # x has the shapw (16,22,3,28,28)

    
    x1 = self.encoder(x)
    # x1 has the shape (16,4,1,1,1)

    
    x2 = self.decoder(x1)
    # x2 has the shape (16,22,3,28,28)

    return x1, x2
</code></pre>
<p>Train:</p>
<p>def train(model, trainloader, criterion, optimizer, epoch):</p>
<pre><code>model.train()

    

for batch_idx, inputs in enumerate(trainloader):
    

    inputs = inputs.float()

    

    if torch.cuda.is_available():

        inputs = inputs.to("cuda")

    optimizer.zero_grad()

    encoded_vectors,outputs = model(inputs)

    loss = criterion(outputs,inputs)

    loss.backward()

    optimizer.step()

    if batch_idx % 50 == 0:

        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(

            epoch, batch_idx * len(inputs), len(trainloader.dataset),

            100. * batch_idx / len(trainloader), loss.item()))
</code></pre>
<p>test:</p>
<p>def test(model, criterion1, testloader):</p>
<pre><code>model.eval()

test_loss = 0

for batch_idx, inputs in enumerate(testloader):

    if torch.cuda.is_available():

        inputs = inputs.cuda()

    encoded_vectors,outputs = model(inputs)

    loss = criterion(outputs,inputs)

    test_loss += loss.item() * inputs.shape[0]

test_loss /= len(testloader.dataset)

print('\nTest set: Average loss: {:.4f}\n'.format(test_loss))

if abs(test_loss) &lt;= 0.005:

    return True

else:

    return False
</code></pre>
<p>def main():</p>
<pre><code>model = Encoder_Decoder()



model = model.to('cuda')

criterion = torch.nn.MSELoss()

optimizer = optim.AdamW(model.parameters(), lr=0.005, eps=1e-3, amsgrad=False)

for epoch in range(20):

    train(model, train_loader, criterion, optimizer, epoch)

    test(model, criterion, test_loader)
   

return model
</code></pre>
<p>if <strong>name</strong> == “<strong>main</strong>”:</p>
<pre><code>model = main()</code></pre>

Preetham_R_Patlolla: <p>I have also tried experimenting different loss functions and different dimensions for the latent space (encoding vector) but nothing worked out and stuck at this part for more than 2 weeks. <a class="mention" href="/u/ptrblck">@ptrblck</a> Kindly address this. Many thanks in advance.</p>

Preetham_R_Patlolla: <p>This is how my loss looks like:</p>
<p>Train Epoch: 0 [0/2000 (0%)]	Loss: 0.890425<br>
Train Epoch: 0 [400/2000 (20%)]	Loss: 0.739331<br>
Train Epoch: 0 [800/2000 (40%)]	Loss: 0.740495<br>
Train Epoch: 0 [1200/2000 (60%)]	Loss: 0.737210<br>
Train Epoch: 0 [1600/2000 (80%)]	Loss: 0.742648</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 1 [0/2000 (0%)]	Loss: 0.738974<br>
Train Epoch: 1 [400/2000 (20%)]	Loss: 0.739232<br>
Train Epoch: 1 [800/2000 (40%)]	Loss: 0.737139<br>
Train Epoch: 1 [1200/2000 (60%)]	Loss: 0.736564<br>
Train Epoch: 1 [1600/2000 (80%)]	Loss: 0.737216</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 2 [0/2000 (0%)]	Loss: 0.737539<br>
Train Epoch: 2 [400/2000 (20%)]	Loss: 0.737013<br>
Train Epoch: 2 [800/2000 (40%)]	Loss: 0.745166<br>
Train Epoch: 2 [1200/2000 (60%)]	Loss: 0.740684<br>
Train Epoch: 2 [1600/2000 (80%)]	Loss: 0.735286</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 3 [0/2000 (0%)]	Loss: 0.738531<br>
Train Epoch: 3 [400/2000 (20%)]	Loss: 0.741940<br>
Train Epoch: 3 [800/2000 (40%)]	Loss: 0.737881<br>
Train Epoch: 3 [1200/2000 (60%)]	Loss: 0.737666<br>
Train Epoch: 3 [1600/2000 (80%)]	Loss: 0.739561</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 4 [0/2000 (0%)]	Loss: 0.741185<br>
Train Epoch: 4 [400/2000 (20%)]	Loss: 0.736491<br>
Train Epoch: 4 [800/2000 (40%)]	Loss: 0.741242<br>
Train Epoch: 4 [1200/2000 (60%)]	Loss: 0.738887<br>
Train Epoch: 4 [1600/2000 (80%)]	Loss: 0.738182</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 5 [0/2000 (0%)]	Loss: 0.738633<br>
Train Epoch: 5 [400/2000 (20%)]	Loss: 0.741287<br>
Train Epoch: 5 [800/2000 (40%)]	Loss: 0.744609<br>
Train Epoch: 5 [1200/2000 (60%)]	Loss: 0.738787<br>
Train Epoch: 5 [1600/2000 (80%)]	Loss: 0.742097</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 6 [0/2000 (0%)]	Loss: 0.737597<br>
Train Epoch: 6 [400/2000 (20%)]	Loss: 0.738081<br>
Train Epoch: 6 [800/2000 (40%)]	Loss: 0.734609<br>
Train Epoch: 6 [1200/2000 (60%)]	Loss: 0.738837<br>
Train Epoch: 6 [1600/2000 (80%)]	Loss: 0.739030</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 7 [0/2000 (0%)]	Loss: 0.742665<br>
Train Epoch: 7 [400/2000 (20%)]	Loss: 0.737820<br>
Train Epoch: 7 [800/2000 (40%)]	Loss: 0.740105<br>
Train Epoch: 7 [1200/2000 (60%)]	Loss: 0.734893<br>
Train Epoch: 7 [1600/2000 (80%)]	Loss: 0.740252</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 8 [0/2000 (0%)]	Loss: 0.740616<br>
Train Epoch: 8 [400/2000 (20%)]	Loss: 0.741398<br>
Train Epoch: 8 [800/2000 (40%)]	Loss: 0.740078<br>
Train Epoch: 8 [1200/2000 (60%)]	Loss: 0.739548<br>
Train Epoch: 8 [1600/2000 (80%)]	Loss: 0.739944</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 9 [0/2000 (0%)]	Loss: 0.739905<br>
Train Epoch: 9 [400/2000 (20%)]	Loss: 0.738651<br>
Train Epoch: 9 [800/2000 (40%)]	Loss: 0.737514<br>
Train Epoch: 9 [1200/2000 (60%)]	Loss: 0.734963<br>
Train Epoch: 9 [1600/2000 (80%)]	Loss: 0.735477</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 10 [0/2000 (0%)]	Loss: 0.736167<br>
Train Epoch: 10 [400/2000 (20%)]	Loss: 0.738089<br>
Train Epoch: 10 [800/2000 (40%)]	Loss: 0.733951<br>
Train Epoch: 10 [1200/2000 (60%)]	Loss: 0.738294<br>
Train Epoch: 10 [1600/2000 (80%)]	Loss: 0.739406</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 11 [0/2000 (0%)]	Loss: 0.739151<br>
Train Epoch: 11 [400/2000 (20%)]	Loss: 0.737162<br>
Train Epoch: 11 [800/2000 (40%)]	Loss: 0.737700<br>
Train Epoch: 11 [1200/2000 (60%)]	Loss: 0.738951<br>
Train Epoch: 11 [1600/2000 (80%)]	Loss: 0.736479</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 12 [0/2000 (0%)]	Loss: 0.738607<br>
Train Epoch: 12 [400/2000 (20%)]	Loss: 0.742146<br>
Train Epoch: 12 [800/2000 (40%)]	Loss: 0.740505<br>
Train Epoch: 12 [1200/2000 (60%)]	Loss: 0.735908<br>
Train Epoch: 12 [1600/2000 (80%)]	Loss: 0.742282</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 13 [0/2000 (0%)]	Loss: 0.736525<br>
Train Epoch: 13 [400/2000 (20%)]	Loss: 0.736685<br>
Train Epoch: 13 [800/2000 (40%)]	Loss: 0.734824<br>
Train Epoch: 13 [1200/2000 (60%)]	Loss: 0.740992<br>
Train Epoch: 13 [1600/2000 (80%)]	Loss: 0.738559</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 14 [0/2000 (0%)]	Loss: 0.735638<br>
Train Epoch: 14 [400/2000 (20%)]	Loss: 0.737805<br>
Train Epoch: 14 [800/2000 (40%)]	Loss: 0.741408<br>
Train Epoch: 14 [1200/2000 (60%)]	Loss: 0.731682<br>
Train Epoch: 14 [1600/2000 (80%)]	Loss: 0.738875</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 15 [0/2000 (0%)]	Loss: 0.740533<br>
Train Epoch: 15 [400/2000 (20%)]	Loss: 0.737641<br>
Train Epoch: 15 [800/2000 (40%)]	Loss: 0.738011<br>
Train Epoch: 15 [1200/2000 (60%)]	Loss: 0.741101<br>
Train Epoch: 15 [1600/2000 (80%)]	Loss: 0.739203</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 16 [0/2000 (0%)]	Loss: 0.741356<br>
Train Epoch: 16 [400/2000 (20%)]	Loss: 0.739178<br>
Train Epoch: 16 [800/2000 (40%)]	Loss: 0.737916<br>
Train Epoch: 16 [1200/2000 (60%)]	Loss: 0.743919<br>
Train Epoch: 16 [1600/2000 (80%)]	Loss: 0.736833</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 17 [0/2000 (0%)]	Loss: 0.739630<br>
Train Epoch: 17 [400/2000 (20%)]	Loss: 0.739462<br>
Train Epoch: 17 [800/2000 (40%)]	Loss: 0.741527<br>
Train Epoch: 17 [1200/2000 (60%)]	Loss: 0.733570<br>
Train Epoch: 17 [1600/2000 (80%)]	Loss: 0.741055</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 18 [0/2000 (0%)]	Loss: 0.739831<br>
Train Epoch: 18 [400/2000 (20%)]	Loss: 0.740010<br>
Train Epoch: 18 [800/2000 (40%)]	Loss: 0.736455<br>
Train Epoch: 18 [1200/2000 (60%)]	Loss: 0.737576<br>
Train Epoch: 18 [1600/2000 (80%)]	Loss: 0.736869</p>
<p>Test set: Average loss: 0.7384</p>
<p>Train Epoch: 19 [0/2000 (0%)]	Loss: 0.739427<br>
Train Epoch: 19 [400/2000 (20%)]	Loss: 0.740225<br>
Train Epoch: 19 [800/2000 (40%)]	Loss: 0.743299<br>
Train Epoch: 19 [1200/2000 (60%)]	Loss: 0.737947<br>
Train Epoch: 19 [1600/2000 (80%)]	Loss: 0.735534</p>
<p>Test set: Average loss: 0.738</p>

Preetham_R_Patlolla: <p>Am I doing some obvious blunder? or Is it possible that my data has no patterns or valid features to learn?</p>

ptrblck: <p>You could start by overfitting a small dataset (e.g. just 10 samples) by playing around with the hyperparameters (optimizer, learning rate etc.).<br>
If that doesn’t help, I would suggest to simplify the model.<br>
Once your can come up with a model and hyperparameters, which overfit this small dataset, you could scale up the use case again by using more data.</p>

Preetham_R_Patlolla: <p>I Have changed the model and then tried as you suggested. Although my efficiency isn’t that great, the technique you have suggested is really a good approach. It helped me a lot.</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>