LBFGS not working on NN, loss not decreasing

Desi20: <p>Hi all, I am trying to compare different optimizer on a NN, however, the L-BFGS algorithm does not work and I don’t know why. The loss is not decreasing and my accuracy is very bad. SGD and Adam do work, so I wonder where my mistake is.</p>
<p>Here is my code:</p>
<blockquote>
<p><span class="hashtag-raw">#Load</span> packages<br>
import torch<br>
import torch.nn as nn<br>
import torchvision.transforms as transforms<br>
import torchvision.datasets as dsets<br>
from torch.autograd import Variable<br>
import torch.nn.functional as F</p>
<p>“”“Load MNIST dataset”“”<br>
train_dataset = dsets.MNIST(root = ‘./data’, train=True,<br>
transform = transforms.ToTensor(),<br>
download = True)<br>
test_dataset = dsets.MNIST(root = ‘./data’, train=False,<br>
transform = transforms.ToTensor())</p>
<p>“”“Make dataset iterable”“”<br>
batch_size=1000</p>
<p>train_loader = torch.utils.data.DataLoader(dataset = train_dataset, batch_size = batch_size,<br>
shuffle = True)</p>
<p>test_loader = torch.utils.data.DataLoader(dataset = test_dataset, batch_size = batch_size,<br>
shuffle = False)</p>
<p>“”“Create model class”“”<br>
class FFN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(FFN, self).<strong>init</strong>()<br>
<span class="hashtag-raw">#Linear</span> functions<br>
self.fc1 = nn.Linear(28*28, 120)<br>
self.fc2 = nn.Linear(120, 84)<br>
self.fc3 = nn.Linear(84, 10)</p>
<pre><code>def forward(self, x):
    out = F.relu(self.fc1(x)) #Non-linearity, can be changed to Tanh,ReLu
    out = F.relu(self.fc2(out))
    #Linear function (readout)
    out = self.fc3(out)
    return out
</code></pre>
<p>“”“Instantiate Model and Optimizer”“”<br>
model = FFN()<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.LBFGS(model.parameters())</p>
<p>“”“Train Model”“”<br>
epochs = 5<br>
for epoch in range(epochs):<br>
for i, (images, labels) in enumerate(train_loader):<br>
<span class="hashtag-raw">#Load</span> images as Variables<br>
images = Variable(images.view(-1, 28*28))<br>
labels = Variable(labels)</p>
<pre><code>    def closure():
        #Clear gradients, not be accumulated
        optimizer.zero_grad()

        #Forward pass to get output
        outputs = model(images)

        #Calculate Loss: softmax + cross entropy loss
        loss = criterion(outputs, labels)

        #Get gradients 
        loss.backward()
        return loss
    
    #update parameters
    optimizer.step(closure)
    
    print('Epoch: {}, Loss: {}'.format(epoch, loss.data[0]))
    
    if (i+1) % 100 == 0:
        
        #Calculate accuracy on testset
        correct = 0 
        total = 0
        #Iterate through test data set
        for images, labels in test_loader:
            #Load images to a Torch Variable
            images = Variable(images.view(-1, 28*28))
            
            #Forward pass only to get output
            outputs = model(images)
            
            #Get prediction
            _, predicted = torch.max(outputs.data,1)
            
            #total number of labels
            total += labels.size(0)
            
            #Total correct predictions
            correct += (predicted ==labels).sum()
        
        accuracy = 100*correct /total
        
        #Print
        print('Epoch: {}, Loss: {}, Accuracy on testset: {}'.format(epoch, loss.data[0], accuracy))
</code></pre>
</blockquote>

ptrblck: <p>Try to lower the learning rate to <code>lr=0.1</code>.<br>
I stopped the training after some iterations and the test accuracy was ~95%.</p>

Desi20: <p>I lowered it in my code, but I still get always the same loss on the training set of 56198.06640625 and my test accuracy is 9.8%. Increasing the batch_size e.g. does not help neither.</p>

ptrblck: <p>That’s strange. I just lowered the learning rate to <code>0.1</code> and returned the loss, since it was missing in your code: <code>loss = optimizer.step(closure)</code>.</p>
<p>Maybe I was lucky? I’ll try it with different seeds to check if the approach is reliable.</p>

ptrblck: <p>I tested it with different seeds and it seems to work:</p>
<pre><code class="lang-auto">Epoch: 0, Loss: 2.30485224724
Epoch: 0, Loss: 1.00339114666
Epoch: 0, Loss: 0.672091126442
Epoch: 0, Loss: 0.501866996288
...
Epoch: 0, Loss: 0.158133521676
Test accuracy: 95%
</code></pre>
<p>Which PyTorch version are you using?</p>

Desi20: <p>True, I was missing that loss = optimizer.step(closure). But I still get</p>
<p>Epoch: 0, Loss: 2.307893753051758<br>
Epoch: 0, Loss: 2.3076794147491455<br>
Epoch: 0, Loss: nan<br>
Epoch: 0, Loss: nan</p>
<p>and the accuracy of 9.8%. It’s super strange that it is working for you and not for me.</p>
<p>And I’m using version 0.3.1.post2, so it’s up-to date.</p>

ptrblck: <p>On my main machine I’ve compiled from master, so I installed <code>0.3.1.post2</code> on another machine and got the same results.</p>
<p>Could you please run <a href="https://gist.github.com/ptrblck/25197491711e38f0d9cb72fe2d9a3f2a">this code</a> on your machine and report the results?</p>

Desi20: <p>I copied your code and only had to change root to root = ‘./data’ because otherwise the access was denied.</p>
<p>But I still get<br>
Epoch: 0, Loss: 2.304852247238159<br>
Epoch: 0, Loss: 2.308029890060425<br>
Epoch: 0, Loss: nan<br>
Epoch: 0, Loss: nan<br>
Epoch: 0, Loss: nan<br>
Epoch: 0, Loss: nan<br>
Epoch: 0, Loss: nan<br>
Epoch: 0, Loss: nan<br>
Epoch: 0, Loss: nan</p>
<p>And I really don’t get it because SGD and ADAM work perfectly fine. And if it is working for you, why not for me.</p>

ptrblck: <p>Ok, that’s strange. Maybe someone else could try to reproduce this issue?</p>

Desi20: <p>I found the mistake, I had Python 3.5 installed in Anaconda and I needed Python 3.6.</p>

RFoldes: <p>Hi,</p>
<p>I am sorry to re-open an old post, but this script puzzles me a bit.</p>
<p>Based on what I read about lbfgs, it should only work with all data at the same time, whereas as far as I can see in this example it is employed using mini-batches.</p>
<p>In this way, it is not updating the network parameters after each mini-batch instead of of after the entire dataset has been processed?</p>
<p>Thanks.</p>