New RL library, for PyTorch!

iffiX: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b8e78246fe737117d5ea114f7d5d9c518d00bb8.png" data-download-href="https://discuss.pytorch.org/uploads/default/2b8e78246fe737117d5ea114f7d5d9c518d00bb8" title="icon_title"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b8e78246fe737117d5ea114f7d5d9c518d00bb8_2_344x94.png" alt="icon_title" data-base62-sha1="6djLL4m1J9Uelfc91S2ZCmjSaEw" width="344" height="94" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b8e78246fe737117d5ea114f7d5d9c518d00bb8_2_344x94.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b8e78246fe737117d5ea114f7d5d9c518d00bb8_2_516x141.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b8e78246fe737117d5ea114f7d5d9c518d00bb8_2_688x188.png 2x" data-dominant-color="5E0000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">icon_title</span><span class="informations">1279×352 19.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Greetings everyone! I am happy to announce that my RL library, <strong>Machin</strong>, <strong>designed for PyTorch</strong>, is close to its first public debut after several months of hard development!</p>
<p>Machin is designed with the elegant torch style in mind, while aiming to cover most of the functions provided by <a href="https://github.com/ray-project/ray" rel="noopener nofollow ugc">Ray</a>. It is hosted at <a href="https://github.com/iffiX/machin" class="inline-onebox" rel="noopener nofollow ugc">GitHub - iffiX/machin: Reinforcement learning library(framework) designed for PyTorch, implements DQN, DDPG, A2C, PPO, SAC, MADDPG, A3C, APEX, IMPALA ...</a></p>
<p>Currently, Machin is at this development stage:</p>
<pre><code class="lang-auto">alpha-1: laying down a framework for all code
alpha-2: finish testing all modules                       
alpha-3: clean and update old examples               
alpha-3.5: add architecture docs, tutorial docs, etc.
beta-1: public tests, collect response and perfect design    &lt; Machin is at here
first official release!
</code></pre>
<h2><a name="algorithms-1" class="anchor" href="#algorithms-1"></a>Algorithms</h2>
<p>Machin is able to support a variety of RL algorithms, including:</p>
<h4><a name="single-agent-algorithms-2" class="anchor" href="#single-agent-algorithms-2"></a>Single agent algorithms:</h4>
<ul>
<li><a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="noopener nofollow ugc">Deep Q-Network (DQN)</a></li>
<li><a href="https://arxiv.org/pdf/1509.06461.pdf" rel="noopener nofollow ugc">Double DQN</a></li>
<li><a href="https://arxiv.org/abs/1511.06581" rel="noopener nofollow ugc">Dueling DQN</a></li>
<li><a href="https://arxiv.org/abs/1710.02298" rel="noopener nofollow ugc">RAINBOW</a></li>
<li><a href="https://arxiv.org/pdf/1509.02971.pdf" rel="noopener nofollow ugc">Deep Deterministic policy Gradient (DDPG)</a></li>
<li><a href="https://arxiv.org/pdf/1802.09477.pdf" rel="noopener nofollow ugc">Twin Delayed DDPG (TD3)</a></li>
<li><a href="https://hal.archives-ouvertes.fr/hal-00187279/document" rel="noopener nofollow ugc">Hystereric DDPG (Modified from Hys-DQN)</a></li>
<li><a href="https://openai.com/blog/baselines-acktr-a2c/" rel="noopener nofollow ugc">Advantage Actor-Critic (A2C)</a></li>
<li><a href="https://arxiv.org/pdf/1707.06347.pdf" rel="noopener nofollow ugc">Proximal Policy Optimization (PPO)</a></li>
<li><a href="https://arxiv.org/pdf/1812.05905.pdf" rel="noopener nofollow ugc">Soft Actor Critic (SAC)</a></li>
</ul>
<h4><a name="multi-agent-algorithms-3" class="anchor" href="#multi-agent-algorithms-3"></a>Multi-agent algorithms:</h4>
<ul>
<li><a href="https://arxiv.org/pdf/1706.02275.pdf" rel="noopener nofollow ugc">Multi-agent DDPG (MADDPG)</a></li>
</ul>
<h4><a name="massively-parallel-algorithms-4" class="anchor" href="#massively-parallel-algorithms-4"></a>Massively parallel algorithms:</h4>
<ul>
<li><a href="https://arxiv.org/abs/1602.01783" rel="noopener nofollow ugc">Asynchronous A2C (A3C)</a></li>
<li><a href="https://arxiv.org/pdf/1803.00933" rel="noopener nofollow ugc">APEX-DQN, APEX-DDPG</a></li>
<li><a href="https://arxiv.org/pdf/1802.01561" rel="noopener nofollow ugc">IMPALA</a></li>
<li><a href="https://arxiv.org/pdf/1803.07055.pdf" rel="noopener nofollow ugc">Augmented random search (ARS, non-gradient)</a></li>
</ul>
<h4><a name="enhancements-5" class="anchor" href="#enhancements-5"></a>Enhancements:</h4>
<ul>
<li><a href="https://arxiv.org/pdf/1511.05952.pdf" rel="noopener nofollow ugc">Prioritized Experience Replay (PER)</a></li>
<li><a href="https://arxiv.org/pdf/1506.02438.pdf" rel="noopener nofollow ugc">Generalized Advantage Estimation (GAE)</a></li>
<li><a href="https://arxiv.org/pdf/1507.06527.pdf" rel="noopener nofollow ugc">Recurrent networks in DQN, etc.</a></li>
</ul>
<h4><a name="algorithms-to-be-supported-6" class="anchor" href="#algorithms-to-be-supported-6"></a>Algorithms to be supported:</h4>
<ul>
<li><a href="https://arxiv.org/abs/1606.03476" rel="noopener nofollow ugc">Generative Adversarial Imitation Learning (GAIL)</a></li>
<li>Evolution Strategies</li>
<li>Linear Optimization Methods</li>
<li><a href="https://arxiv.org/abs/1803.11485" rel="noopener nofollow ugc">QMIX (multi agent)</a></li>
<li>Model-based methods</li>
</ul>
<h2><a name="parallel-7" class="anchor" href="#parallel-7"></a>Parallel</h2>
<p>Machin is capable of:</p>
<ol>
<li>Distributed &amp; asynchronus training: provide implementations for parameter servers and gradient reduction servers.</li>
<li>Advance RPC: Role based launching, service registraction, resource sharing. constructed on the basis of torch RPC.</li>
<li>Process level Parallelization, based on enhanced Processes, Threads, Pools, Events.</li>
<li>Model level Parallelization, Splitting(Not yet implemented) and Assignning model shards to devices, by heuristic.</li>
</ol>
<h2><a name="utilities-8" class="anchor" href="#utilities-8"></a>Utilities</h2>
<ol>
<li>Model checking toolset: check for nan values, abnormal gradients etc in your model, based on hooks.</li>
<li>Logging videos, images, produced during your training</li>
<li>Model serialization, auto loading and management toolset. No longer worry about unmatched devices!</li>
<li>Many more.</li>
</ol>

iffiX: <p>I wonder if there is any specific need for Model based methods? I am quite interested in this area, and wants to hear your suggestions on:</p>
<ol>
<li>What model-based algorithms are STOA and worth supporting.</li>
<li>Is there a good overview paper for this ?</li>
</ol>