There are a Leverberg-Marquardt like optimizer for pytorch?

rerere_L: <p>I have a non linear regression NN, i wanted know if there are a optimizer Leverberg-Marquardt like  that can i use in my case?</p>
<pre><code class="lang-auto">N, D_in, H, D_out = x.shape[0], x.shape[1], 6, y.shape[1]

model = nn.Sequential(OrderedDict([ ('fc1', nn.Linear(D_in, H)), 
                                    #('Sig', nn.Sigmoid()),
                                    ('ISRU', ISRU()), # Add ISRU
                                    ('fc2', nn.Linear(H, D_out))]))

# Error -----
loss_fn = torch.nn.L1Loss(reduction='mean')

# Train -----
optimizer = *****
</code></pre>

crowsonkb: <p>I tried Googling it and I couldn’t find any implementations of that optimizer for PyTorch. Most of what exists is variations on first-order gradient descent. If your gradients are not stochastic you might try to use <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.LBFGS" rel="nofollow noopener">torch.optim’s implementation</a> of the second-order optimizer <a href="https://en.wikipedia.org/wiki/Limited-memory_BFGS" rel="nofollow noopener">L-BFGS</a> (be sure to set <code>line_search_fn='strong_wolfe'</code> or you risk the optimizer ‘blowing up’ due to accepting a step which increases the loss).</p>

rerere_L: <p>an Example for train loop with this optimizer?</p>

crowsonkb: <p>Here’s an example of minimizing the <a href="https://en.wikipedia.org/wiki/Rosenbrock_function" rel="nofollow noopener">Rosenbrock function</a> with L-BFGS:</p>
<pre><code class="lang-python">from functools import partial

import torch
from torch import optim


def rosenbrock(x):
    return (1 - x[0])**2 + 100*(x[1] - x[0]**2)**2


a = torch.tensor([1., 1.]) + torch.rand(2)
a.requires_grad_()
loss = partial(rosenbrock, a)


def print_iter(i, a, loss):
    print(f'{i} [{a[0]:.6f}, {a[1]:.6f}], loss: {loss:.6f}')


opt = optim.LBFGS([a], line_search_fn='strong_wolfe')
print_iter(0, a, loss())
for i in range(200):
    opt.zero_grad()
    loss().backward()
    opt.step(loss)
    print_iter(i+1, a, loss())
</code></pre>

rerere_L: <p>in the case i want use nn.MSELoss, who to change this axample ?</p>

fabiodimarco: <p>I know this thread is a bit old, but for anyone still looking for a Levenberg-Marquardt implementation in PyTorch, I’ve developed one: <a href="https://github.com/fabiodimarco/torch-levenberg-marquardt" rel="noopener nofollow ugc">torch-levenberg-marquardt</a>. Hope it helps anyone who comes across this!</p>

Robert_Barrios: <p>Hi Fabio. In this link <a href="https://github.com/pytorch/pytorch/issues/83529" rel="noopener nofollow ugc">Adding Levenberg-marquardt optimizer in PyTorch · Issue #83529 · pytorch/pytorch</a> says that implementing pytorch tensors is very slow for training. Do you think that your implementation solved that problem?</p>

fabiodimarco: <p>Hi Robert, I think my implementation doesn’t suffer from the issues mentioned in that discussion. Using jacrev + vmap to compute the Jacobian in PyTorch is fast and memory-efficient, especially compared to other methods I’ve tried in both PyTorch and TensorFlow.<br>
Of course, you cannot expect to train models with billions of parameters using LM, but for certain architectures, I’ve been able to train models with millions of parameters on a GPU.</p>