Ppo+lstm working code

granth_jain: <p>Hi,</p>
<p>I am looking for ppo + lstm implementation.<br>
Can someone please help to let me know of available working code in pytorch for ppo + lstm.</p>
<p>Thanks</p>

EsraaElelimy: <p>Hi,<br>
I am not sure if it’s too late to answer this but I came across this implementation for ppo with lstm : <a href="https://github.com/seungeunrho/minimalRL/blob/master/ppo-lstm.py" rel="noopener nofollow ugc">https://github.com/seungeunrho/minimalRL/blob/master/ppo-lstm.py</a><br>
and the code is quite simple and easy to follow.<br>
Hope it helps.</p>

leckofunny: <p>Hi <a class="mention" href="/u/granth_jain">@granth_jain</a><br>
did you find a suitable implementation.</p>
<p>Unfortunately, the one proposed before is not really a good choice. It uses truncated bptt of sequence length 1. The CartPole environment is not a good environment to test if the recurrent policy is working even if you mask out the velocities of the agent’s observation space.</p>

THEKROLL: <p>Not exactly the wanted solution but a working (not another cartPole project) LSTM implementation. When you increase the sequence_length we feed the model and provide a more complex rewarding in the step function you can test how the model learns to remember sequences and relations in the past: <a href="https://github.com/svenkroll/simple_RL-LSTM" class="inline-onebox" rel="noopener nofollow ugc">GitHub - svenkroll/simple_RL-LSTM: A simple demonstration of how to train an LSTM model with Reinforcement Learning using PyTorch</a></p>

leckofunny: <p>Clean baseline repositories</p>
<p>PPO + LSTM (or GRU)</p>
<p><a>https://github.com/MarcoMeter/recurrent-ppo-truncated-bptt</a></p>
<p>PPO + Transformer-XL</p>
<p><a>https://github.com/MarcoMeter/episodic-transformer-memory-ppo</a></p>

vmoens: <p>This is entirely supported in torchrl.<br>
Here’s an example with DQN</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/rl/blob/1083b35ef9733b2335bd88d587cb282e180267c4/tutorials/sphinx-tutorials/dqn_with_rnn.py#L242">
  <header class="source">

      <a href="https://github.com/pytorch/rl/blob/1083b35ef9733b2335bd88d587cb282e180267c4/tutorials/sphinx-tutorials/dqn_with_rnn.py#L242" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/rl/blob/1083b35ef9733b2335bd88d587cb282e180267c4/tutorials/sphinx-tutorials/dqn_with_rnn.py#L242" target="_blank" rel="noopener nofollow ugc">pytorch/rl/blob/1083b35ef9733b2335bd88d587cb282e180267c4/tutorials/sphinx-tutorials/dqn_with_rnn.py#L242</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="232" style="counter-reset: li-counter 231 ;">
          <li>#   However, to respect TorchRL's conventions, this LSTM must have the ``batch_first``</li>
          <li>#   attribute set to ``True`` which is **not** the default in PyTorch. However,</li>
          <li>#   our :class:`~torchrl.modules.LSTMModule` changes this default</li>
          <li>#   behavior, so we're good with a native call.</li>
          <li>#</li>
          <li>#   Also, the LSTM cannot have a ``bidirectional`` attribute set to ``True`` as</li>
          <li>#   this wouldn't be usable in online settings. In this case, the default value</li>
          <li>#   is the correct one.</li>
          <li>#</li>
          <li></li>
          <li class="selected">lstm = LSTMModule(</li>
          <li>    input_size=n_cells,</li>
          <li>    hidden_size=128,</li>
          <li>    device=device,</li>
          <li>    in_key="embed",</li>
          <li>    out_key="embed",</li>
          <li>)</li>
          <li></li>
          <li>######################################################################</li>
          <li># Let us look at the LSTM Module class, specifically its in and out_keys:</li>
          <li>print("in_keys", lstm.in_keys)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I know some folks have open source code with PPO and LSTM/GRU<br>
You will find real-life examples in this repo for instance: <a href="https://github.com/Acellera/acegen-open" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Acellera/acegen-open: Language models for drug discovery using torchrl</a></p>
<p>Happy to provide more context if needed!</p>