Implementing new Optimization algorithms

username: bhushans23
cooked: <p>I am looking forward to implement  ‘<a href="https://epubs.siam.org/doi/abs/10.1137/15M1053141" rel="nofollow noopener">Quasi Newton algorithm</a>’  in pytorch.</p>
<p>Looking for suggestions of other optimization algorithms that you think are useful?  I will go through and implement bunch of it.</p>

username: tom
cooked: <p>Last year, quite a few people talked about KFAC, but it might not be too easy to fit into the pytorch optimizer framework.<br>
One of the most prominent implementations seems to be this:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/yaroslavvb/kfac_pytorch/blob/master/kfac_pytorch.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/yaroslavvb/kfac_pytorch/blob/master/kfac_pytorch.py" target="_blank" rel="nofollow noopener">yaroslavvb/kfac_pytorch/blob/master/kfac_pytorch.py</a></h4>
<pre><code class="lang-py">import util as u
u.check_mkl()

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import numpy as np
import scipy
import sys

from torch.autograd.function import Function

def regularized_inverse(mat, lambda_=3e-3, inverse_method='numpy',
                        use_cuda=True):
  assert mat.shape[0] == mat.shape[1]
  ii = torch.eye(mat.shape[0])
  if use_cuda:
    ii = ii.cuda()
</code></pre>

  This file has been truncated. <a href="https://github.com/yaroslavvb/kfac_pytorch/blob/master/kfac_pytorch.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Best regards</p>
<p>Thomas</p>

username: Filos92
cooked: <p>I think for Regression Problems the Levenberg Marquardt Algorithmus is quite usefull!</p>

username: wqsgsutd
cooked: <p>HI, Filos<br>
Do you know how to implement levenberg marquardt in pytorch?</p>
<p>Thanks !</p>

username: Filos92
cooked: <p>You mean LM for general regression or as a substitute for SGD? Last one No, I’m sorry</p>

username: wqsgsutd
cooked: <p>Just LM for general regression. Do you have an example? Thanks!</p>

username: fabiodimarco
cooked: <p>For anyone still looking for a Levenberg-Marquardt implementation in PyTorch, I’ve developed one:</p>
<aside class="onebox githubrepo" data-onebox-src="https://github.com/fabiodimarco/torch-levenberg-marquardt">
  <header class="source">

      <a href="https://github.com/fabiodimarco/torch-levenberg-marquardt" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">
  <img width="690" height="344" src="https://opengraph.githubassets.com/2a5dcb3f01117476eb6fb13a5898e829/fabiodimarco/torch-levenberg-marquardt" class="thumbnail">

  <h3><a href="https://github.com/fabiodimarco/torch-levenberg-marquardt" target="_blank" rel="noopener nofollow ugc">GitHub - fabiodimarco/torch-levenberg-marquardt: PyTorch implementation of Levenberg-Marquardt...</a></h3>

    <p><span class="github-repo-description">PyTorch implementation of Levenberg-Marquardt training algorithm</span></p>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Hope it helps.</p>