Dynamic mixed optimization using SGD and L-BFGS

basilique: <p>Hi, I am using Pytorch to reproduce the results of the 1D linear advection problem in <a href="https://arxiv.org/abs/1711.06464" rel="noopener nofollow ugc">this paper</a>. The problem is quite simple. I am not however able to reproduce the results using the given settings : a fully connected NN with 2 hidden layers each with 10 hidden units, sigmoid activation function and <strong>a mixed dynamic training using both SGD and L-BFGS optimizers</strong>:</p>
<blockquote>
<p>we have implemented the gradients according to the schemes presented in the<br>
previous section and we consequently use the BFGS method<br>
with default settings from SciPy to train the ANN.</p>
<p>An issue one might encounter is that the line search in BFGS<br>
fails due to the Hessian matrix being very ill-conditioned. To circumvent this we use a BFGS in combination with stochastic gradient descent. When BFGS fails due to line search failure we run<br>
1000 iterations with stochastic gradient descent with a very small learning rate, in the order of 10−9, to bring BFGS out of the troublesome region. This procedure is repeated until convergence or until the maximum number of allowed iterations has been exceeded</p>
</blockquote>
<p>I would like to know how I can implement their dynamic training approach in Pytorch. I just managed to first pre train the model using SGD for <code>n</code> iterations and then train it using L-BFGS until it converges. something like:</p>
<pre><code class="lang-auto">def loss_LBFGS(self):
    
    # operations to calculate the loss

    self.optimizer_LBFGS.zero_grad()
    loss.backward()
    return loss


def train(self, n_iteration_sgd):
    self.ann_u.train()

    # *****************************************************************************
    # pretrain with the SGD optimizer
    for i in range(n_iteration_sgd):
        
        # the same operations to calculate the loss

        self.optimizer_sgd.zero_grad()
        loss.backward()
        self.optimizer_sgd.step()
    # *****************************************************************************

    # training using the LBFGS optimizer
    self.optimizer_LBFGS.step(self.loss_LBFGS)
</code></pre>
<p>But this is not exactly what they do. The results that I get using my <code>train</code> function are shown in the following figure:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac30413a2df8a2bd0a52caeee66efdfa2d963ed2.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac30413a2df8a2bd0a52caeee66efdfa2d963ed2" title="n_pde_100_M_10_L_2_n_pde_100_n_adam_100_adam_1e-09_LBFGS_1.0"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac30413a2df8a2bd0a52caeee66efdfa2d963ed2_2_690x429.png" alt="n_pde_100_M_10_L_2_n_pde_100_n_adam_100_adam_1e-09_LBFGS_1.0" data-base62-sha1="ozfvDJuGegcw4OmTskbsOZmuGwq" width="690" height="429" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac30413a2df8a2bd0a52caeee66efdfa2d963ed2_2_690x429.png, https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac30413a2df8a2bd0a52caeee66efdfa2d963ed2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac30413a2df8a2bd0a52caeee66efdfa2d963ed2.png 2x" data-dominant-color="FCFAFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">n_pde_100_M_10_L_2_n_pde_100_n_adam_100_adam_1e-09_LBFGS_1.0</span><span class="informations">770×479 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I think the problem comes from my training approach that is not fully correct. I would appreciate any help. Thank you.</p>