How to demonstrate (or teach) learning theory using PyTorch

AjayTalati: <p>Hi,</p>
<p>I was wondering if any repo’s or notebooks are in the eco-system which use PyTorch to demonstrate <strong>theory</strong>, rather than do experiments, (although the border between the two is a bit vague).</p>
<p>There’s a lot of beautiful theory on function approximation, for example,</p>
<p><a href="https://pdfs.semanticscholar.org/3038/fa7d3cde1f2ecfc41bc6c0c707fe4afd52c1.pdf" rel="nofollow noopener">Learning Real and Boolean Functions: When Is Deep Better Than Shallow</a></p>
<p>is quite readable.</p>
<p>I think it builds on <a href="https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_theory" rel="nofollow noopener">Vapnik–Chervonenkis theory</a>.</p>
<p><a class="mention" href="/u/smth">@smth</a> - Vladimir Vapnik is at FAIR now, (I think)? So I guess there might be some, “in house teaching courses”, that covers statistical learning theory, and is implemented using PyTorch?</p>
<p><a class="mention" href="/u/apaszke">@apaszke</a>, <a class="mention" href="/u/fmassa">@fmassa</a> is this something you guys would be interested in? Or, is the focus of PyTorch more leaning towards experiments/applications?</p>

AjayTalati: <p>If anyone knows of any good <strong>theory</strong> repos/projects that are in any deep learning framework, (doesn’t have to be PyTorch). Please could you link them here?</p>
<p>I’d be interested in implementing them in PyTorch - I’ll be doing some DL theory teaching/demos over the summer.</p>
<p>This would also be very helpful to others who are giving DL demos/teaching.</p>
<p>Thanks a lot for your help,</p>
<p>Aj</p>

smth: <p>definitely not learning theory, but a notebook that teaches NLP properly using pytorch as a tool, rather than showcase pytorch as a tech-demo was this:<br>
<a href="https://github.com/rguthrie3/DeepLearningForNLPInPytorch" class="onebox" target="_blank">https://github.com/rguthrie3/DeepLearningForNLPInPytorch</a></p>
<p>In general, to give you my completely honest and regularized opinion, teaching something like “Learning Real and Boolean Functions” is way more valuable to teach in numpy than in pytorch. I wont pretend, numpy is much more accessible.</p>
<p>Unless you need to really teach / showcase a gradient based learning method (SVM-SGD?) or teach GPU based whatever, you can maybe just stick to numpy.</p>

tom: <p>On my list of things I’d like to see or do is an implementation of<br>
<a href="https://arxiv.org/abs/1612.00410" rel="nofollow noopener">Alemi et al.: Deep Variational Information Bottleneck</a><br>
or<br>
<a href="https://arxiv.org/abs/1703.00810" rel="nofollow noopener">Shwartz-Ziv and Tishby: Opening the Black Box of Deep Neural Networks via Information</a></p>
<p>Also, I think the Bayesian neural network chapters in the textbook<br>
<a href="http://www.inference.phy.cam.ac.uk/itprnn/book.pdf" rel="nofollow noopener">MacKay: Information Theory, Inference, and Learning Algorithms</a><br>
might be inspiration for some things you could do. Though for this, Soumith’s comment regarding numpy vs. torch may apply as well.</p>
<p>Naturally, I’d be very interested in what you do. <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Best regards</p>
<p>Thomas</p>

Bhavya_Bhatt: <p>I know this is an old conversation but I am developing a full fledged package for information theory of deep learning in PyTorch which have a lot of information bottleneck functionalities including HSIC bottleneck sigma networks (yes ! that train without backprop). Currently library is in testing phase but there are notebooks available, thought that can be useful.<br>
Link to the repo:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/spino17/PyGlow" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/43537018?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/spino17/PyGlow" target="_blank" rel="nofollow noopener">spino17/PyGlow</a></h3>

<p>This package is an attempt to implement Keras like API functionalities on PyTorch backend with functionalities supporting information theoretic methods which are relevant for understanding neural n...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Link to the documentation:<br>
<a href="https://pyglow.github.io/" class="onebox" target="_blank" rel="nofollow noopener">https://pyglow.github.io/</a></p>