Using PyTorch optimizers for nonlinear least squares curve fitting

Thomas_Aarholt: <p>Preface: This question is not about training neural nets to perform curve fitting. I’m wondering about using the optimizers to perform minimization for curve fitting, with the aim of eventually moving calculations to the GPU.</p>
<p>Hi! I’m relatively new to using PyTorch. I’m wishing to use the pytorch’s optimizers with automatic differentiation in order to perform nonlinear least squares curve fitting. Since the Levenberg–Marquardt algorithm doesn’t appear to be implemented, I’ve used the L-BFGS optimizer. They both take advantage of second-order derivatives which PyTorch supports doing.</p>
<p>My question is how to improve the following code - in particular, I’m wondering about what I should be thinking about regarding the learning rate - the data fed to the optimizer is “smaller than memory” - somewhere between 2000 and 1 million data points.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.functional import F

def gaussian(x, A=1, sigma=1, centre=0):
    return A*torch.exp(-(x - centre)**2 / (2*sigma**2))

def func(x, weights):
    'Takes a tensor of three sets of gaussian parameters, returns the sum of three gaussians'
    return gaussian(x, *weights[:3]) + gaussian(x, *weights[3:6]) + gaussian(x, *weights[6:9])

params = torch.tensor([0.8, 0.5, -1.5, 1, 2, 0, 3, 0.4, 1.8])
x = torch.linspace(-10, 10, 10000)
target = func(x, params)

guess = [1, 1, -0.2, 1, 1, 0.0, 1, 1, 0.2]
weights_LBFGS = torch.tensor(guess, requires_grad=True)
weights = weights_LBFGS

optimizer = torch.optim.LBFGS([weights_LBFGS], max_iter=10000, lr=0.1)
guesses = []
losses = []
def closure():
    optimizer.zero_grad()
    output = func(x, weights)
    loss = F.mse_loss(output, target)
    loss.backward()
    guesses.append(weights.clone())
    losses.append(loss.clone())
    return loss

%time optimizer.step(closure) # %time is jupyter timing magic - remove it to run this outside ipython
print(f"Minimum: {[v.item() for v in weights]}")
print(f"Number of steps: {len(guesses)}")

# Wall time: 683 ms
# Minimum: [0.8001920580863953, 0.4998912513256073, -1.4998143911361694, 0.9999790787696838, 2.0001237392425537, 0.00012422756117302924, 2.9998154640197754, 0.4000047743320465, 1.7999995946884155]
# Number of steps: 256
</code></pre>
<p>The fit works, but it takes time - about ten times what scipy does using <code>scipy.optimize.curve_fit</code>. If I move it onto the GPU (by adding <code>device='cuda'</code> to the <code>tensor</code> functions), it takes a further 10 times as long! (5 sec)</p>
<pre data-code-wrap="python"><code class="lang-python">import numpy as np
from scipy.optimize import curve_fit

def gaussian_numpy(x, A=1, sigma=1, centre=0):
    return A*np.exp(-(x - centre)**2 / (2*sigma**2))

def func_numpy(x, *weights):
    return gaussian_numpy(x, *weights[:3]) + gaussian_numpy(x, *weights[3:6]) + gaussian_numpy(x, *weights[6:9])

xi = x.numpy()
yi = target.numpy()

%time fit, err = curve_fit(func_numpy, xi, yi, p0=guess, method='lm')
</code></pre>
<p>Both cases produce a good fit, but I would need to improve the speed before I can consider using it as an alternative to scipy. I have played with some other optimizers (Adam) but it seems to perform rather poorly for this - I can go more into detail on that in a reply below.</p>
<pre data-code-wrap="python"><code class="lang-python"># Visualising the fit
import matplotlib.pyplot as plt
plt.figure(dpi=150)
plt.plot(xi, yi, label='target')
plt.plot(xi, func_numpy(xi, *fit), ls='--', color='red', label='scipy')
plt.plot(xi, func(x, weights_LBFGS).detach(), ls='--', color='green', label='torch')
plt.legend()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/903669dd9f7b07a7d569db1e41e64295cb79e723.png" data-download-href="https://discuss.pytorch.org/uploads/default/903669dd9f7b07a7d569db1e41e64295cb79e723" title="fit"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/903669dd9f7b07a7d569db1e41e64295cb79e723_2_690x459.png" alt="fit" data-base62-sha1="kzLkQoKhntBMcNGn5yp1A2IFiJZ" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/903669dd9f7b07a7d569db1e41e64295cb79e723_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/903669dd9f7b07a7d569db1e41e64295cb79e723_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/0/903669dd9f7b07a7d569db1e41e64295cb79e723.png 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fit</span><span class="informations">1200×800 41.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>