Implementation of Elastic Weight Consolidation / Fisher information matrix

nigel: <p>Hey guys,</p>
<p>I was wondering if anyone has implemented Elastic Weight Consolidation (EWC) as outlined in <a href="https://arxiv.org/abs/1612.00796" rel="nofollow noopener">this paper</a>? This algorithm allows for sequential/continuous learning without the model encountering catastrophic forgetting.</p>
<p>The main part of implementing this is calculating the Fisher information matrix. If anyone has any code they can share on this, that’d be great. Otherwise I’m happy to attempt it and share my code here.</p>
<p>Found a tensorflow implementation here: <a href="https://github.com/stokesj/EWC" rel="nofollow noopener">https://github.com/stokesj/EWC</a> which we can use for reference.</p>

Prashnna_K_Gyawali: <p>Did you find/do anything on PyTorch?</p>

pkadambi: <p>These two repos might have what you’re looking for:</p>
<p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/moskomule/ewc.pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/11806234?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/moskomule/ewc.pytorch" target="_blank" rel="nofollow noopener">moskomule/ewc.pytorch</a></h3>

<p>An implementation of EWC with PyTorch. Contribute to moskomule/ewc.pytorch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/kuc2477/pytorch-ewc" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/9589618?s=400&amp;v=4" class="thumbnail onebox-avatar" width="220" height="220">

<h3><a href="https://github.com/kuc2477/pytorch-ewc" target="_blank" rel="nofollow noopener">kuc2477/pytorch-ewc</a></h3>

<p>PyTorch implementation of DeepMind's PNAS 2017 paper "Overcoming Catastrophic Forgetting" - kuc2477/pytorch-ewc</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

rusty: <p>Hello. Are you that egg fried rice guy?</p>

acobobby: <p>The original EWC requires you to compute the importance for each weight based on an additional pass over the training set. The importance is the squared gradient averaged over each minibatch. Anyway, you can take a look at the implementation available in the <a href="https://github.com/ContinualAI/colab/blob/master/notebooks/intro_to_continual_learning.ipynb" rel="noopener nofollow ugc">ContinualAI notebooks</a>. It is an association for Continual Learning <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Disclaimer: I am part of ContinualAI.</p>