Validation Loss not Decreasing for Autoencoder

username: rtkaratekid
cooked: <p>Finally got fed up with tensorflow and am in the process of piping a project over to pytorch. So far I’ve found pytorch to be different but MUCH more intuitive.</p>
<p>One of my nets is a good old fashioned autoencoder I use for anomaly detection of unlabelled data. I’ve set it up to periodically report my current training and validation loss and have come across a head scratcher. My training loss improves about what I’d expect (although faster would be great), but my validation loss remains essentially the same. I’ve perused the forums here and can’t find anything that helps. Admittedly, while I can build nets pretty well in tensorflow, this could just be a stupid error on my part. I know it isn’t the data as the same data, formatted exactly the same way performs well in tensorflow. And I have… a lot of data. So no issues there. I’ll post a sample below of the minimum working code that reproduces the error.</p>
<p>If you want machine specs, I’d be happy to post if they’re relevant (three gpus).</p>
<p>Thanks!</p>
<p>Formatting data and defining the autoencoder</p>
<pre><code class="lang-python">train_data = torch.from_numpy(df[fullData].values[:train])
val_data = torch.from_numpy(df[fullData].values[validate:])

batchsize = 1024
train_iter = DataLoader(dataset=train_data, batch_size=batchsize, shuffle=True)
val_iter = DataLoader(dataset=val_data, batch_size=batchsize, shuffle=True)

class Model(nn.Module):

    def __init__(self, input_size, output_size, droprate):
        super(Model, self).__init__()            
        self.en1 = nn.Linear(input_size, 640)
        self.dp1 = nn.Dropout(droprate)
        self.en2 = nn.Linear(640, 320)
        self.dp2 = nn.Dropout(droprate)
        self.en3 = nn.Linear(320, 160)
        self.dp3 = nn.Dropout(droprate)
        self.en4 = nn.Linear(160, 80)
        self.dp4 = nn.Dropout(droprate)
        self.dec1 = nn.Linear(80, 160)
        self.dp5 = nn.Dropout(droprate)
        self.dec2 = nn.Linear(160, 320)
        self.dp6 = nn.Dropout(droprate)
        self.dec3 = nn.Linear(320, 640)
        self.dp7 = nn.Dropout(droprate)
        self.dec4 = nn.Linear(640, output_size)     

    def forward(self, ins):
        x = F.elu(self.en1(ins))
        x = self.dp1(x)
        x = F.elu(self.en2(x))
        x = self.dp2(x)
        x = F.elu(self.en3(x))
        x = self.dp3(x)
        x = F.elu(self.en4(x))
        x = self.dp4(x)
        
        x = F.elu(self.dec1(x))
        x = self.dp5(x)
        x = F.elu(self.dec2(x))
        x = self.dp6(x)
        x = F.elu(self.dec3(x))
        x = self.dp7(x)
        output = F.elu(self.dec4(x))

        return output
</code></pre>
<p>Aaaaand doing the training</p>
<pre><code class="lang-python">model = Model(input_size, output_size, 0.5)
if torch.cuda.device_count() &gt; 1:
    model = nn.DataParallel(model)
model.to(device)

optimizer = optim.Adam(model.parameters(), lr=0.0001, weight_decay=0.00001)

num_epochs = 10
iters = 0
model = model.double()
validate = 100
best_val_loss = 100
vals = []
losses = []
for epoch in range(num_epochs):
    for batch_idx, batch in enumerate(loader):
        model.train()
        optimizer.zero_grad()
        
        iters += 1
        inputs = batch.to(device)
        output = model(inputs)
        train_loss = criterion(output, inputs)

        train_loss.backward()
        optimizer.step()

        if iters % validate == 0:
            val_loss = 0
            model.eval()

            with torch.no_grad():
                for val in (val_iter):
                    val = val.to(device)
                    answer = model(val)
                    val_loss = criterion(answer, val)

            vals.append(val_loss.item())
            iterations.append(iters)
            losses.append(train_loss.item())
</code></pre>

username: phan_phan
cooked: <p>Each time <code>iters % validate == 0</code>, you append only the <em>last</em> <code>train_loss</code> to <code>lossess</code>, and the <em>last</em> <code>val_loss</code> to <code>vals</code>.<br>
Maybe you’d want to accumulate them, and append their means for example.</p>

username: rtkaratekid
cooked: <p><a class="mention" href="/u/phan_phan">@phan_phan</a> thanks for the reply!<br>
I went ahead and modified my code to accumulate the averages of the values as you suggested. My results look a <strong>little</strong> better but I think maybe more so just confirms the problem further haha</p>
<p>For reference my:<br>
starting training loss was 0.016 and validation was 0.0019,<br>
final training loss was 0.004 and validation loss was 0.0007.</p>
<p>And here’s a viz of the losses over ten epochs of training. Based on this, I think the model is improving and I’m not calculating validation loss correctly, but I can’t figure out anything I’m doing wrong!<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2cd0e9e7b82a0ca8aef359dfb98020119fa36018.png" alt="avg_train_vs_val_loss" data-base62-sha1="6osBsCGxpKICBfpDlrDEjV6715m" width="332" height="216"></p>

username: phan_phan
cooked: <p>What is curious is that the validation loss seems to converge 10 times faster than the training loss.<br>
To better understand what is going on, could you:</p>
<ul>
<li>Do validation at a higher frequency ; for example at <code>validate = 20</code>
</li>
<li>Try to train it without dropout… Just to see if something changes</li>
</ul>
<p><em>By the way, a <code>nn.Dropout()</code> layer has no parameters. So you can define <strong>a single layer</strong> instead of seven : <code>self.dp = nn.Dropout(droprate)</code></em><br>
<em>And in forward call this layer multiple times : <code>x = self.dp(x)</code>.</em></p>

username: rtkaratekid
cooked: <p>It’s funny that you mention that about the dropout because right before reading you comment I had a little facepalm moment when I realized just that.</p>
<p>I also tried taking out the dropout and what happened was the validation accuracy had the same behavior (where it was essentially the same over time), but the accuracy improved to the range of 1e-5.</p>
<p>When I validate more often  (which was a good suggestion) I think it’s uncovered a little more granular view of it not improving haha.</p>
<p><strong>This is validating every 20 without dropout for 10 epochs</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/bac8a9a93a655d0c5393c139679efa6f99f792a6.png" alt="nodrop_val_every_20" data-base62-sha1="qEmJ6gPed48vvy3me84KYDoE5hk" width="332" height="217"></p>
<p><strong>This is the same, but with dropout rate of 0.5 between each layer</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af250538ed9fe937b6182a4efccb794c41a0eac6.png" alt="dropout_val_every_20" data-base62-sha1="oZoS8sL7xkkqUYMGe0BlUJv5WOq" width="335" height="213"></p>
<p>What I might need to do is just copy some other project and try and run it and see if I can replicate their results. The validation scores are good, my worry is just that since they aren’t improving, 1) something is wrong, and 2) that limits how accurate my model can be in the long run</p>

username: rtkaratekid
cooked: <p>Yeah, not sure why, but pirating <a href="https://github.com/L1aoXingyu/pytorch-beginner/blob/master/08-AutoEncoder/simple_autoencoder.py" rel="nofollow noopener">this code</a> got me the kind of performance one would expect from a neural net. I’ll have to look into the differences a little more to understand why it’s working while my model doesn’t.</p>