Synchronous updates for DPPO

alexis-jacq: <p>I am trying to implement Deepmind’s Distributed Proximal Policy Optimization (<a href="https://arxiv.org/abs/1707.02286">https://arxiv.org/abs/1707.02286</a>)</p>
<p>But I am not really confident with multiprocessing, and I don’t see how to realise the synchronous updates. The idea is to have a chief process that collects the gradients sent by the training workers. When the chief receives enough gradients (more than N gradients) it sums them and does the optimizer’s update. The workers that send the gradients have to wait for the chief’s update before continuing their runs. (the paper explains it in a much nicer way in the supplemental section.)</p>
<p>Also, unfortunately, all the hyperparameters are not provided by Google. I have to tune it by myself.</p>
<p>My code is there:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/alexis-jacq/Pytorch-DPPO" target="_blank">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/9195965?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/alexis-jacq/Pytorch-DPPO" target="_blank">alexis-jacq/Pytorch-DPPO</a></h3>

<p>Pytorch implementation of Distributed Proximal Policy Optimization: https://arxiv.org/abs/1707.02286 - alexis-jacq/Pytorch-DPPO</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
So far, I am trying to solve a simple pendulum environment (and it’s not even converging).<br>
Don’t hesitate to contribute, I need help!</p>

ethancaballero: <p>ELF (rlpytorch) provides source code for synchronous updates:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/facebookresearch/ELF/tree/master/rlpytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/16943930?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/facebookresearch/ELF/tree/master/rlpytorch" target="_blank" rel="nofollow noopener">facebookresearch/ELF</a></h3>

<p>An End-To-End, Lightweight and Flexible Platform for Game Research - facebookresearch/ELF</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

ethancaballero: <p>Actually, nvm. ELF is still asynchronous, but it uses batching &amp; bucketing so that it’s less asynchronous than normal A3C.</p>

dgriff: <p>ELF updates look synchronous to me. When used in this context it mean using the same clock so the processes are actually done sequentialially.</p>
<p>You could use that as reference but instead of when they update parameters from queue one by one just change that to how you want the gradients accumulated then updated.</p>
<p>You will want to use a queque to do this synchronous</p>
<p>Checkout the code on this part for refs <a href="https://github.com/facebookresearch/ELF/blob/master/rlpytorch/utils.py" rel="nofollow noopener">https://github.com/facebookresearch/ELF/blob/master/rlpytorch/utils.py</a></p>

ethancaballero: <p><a class="mention" href="/u/dgriff">@dgriff</a> Have you gotten ELF working with gym?<br>
I’m currently trying to modify ELF source code to work with gym.</p>

dgriff: <p>Not sure what you mean ELF suppose to be alternative to gym.</p>
<p>Or do you mean their actor critic example they provide which looks to be the same model as tensorpack example that does gym which has same author so probably is the case. So I would assume same performance as well which is a quite a good implementation. But have no plans on changing this example to run in gym sorry.</p>

alexis-jacq: <p>I think I solved my synchronous problem. I added 3 function to my shared models:</p>
<pre><code class="lang-python">    def cum_grads(self):
        for name, p in self.named_parameters():
            if p.grad is not None:
                val = self.__getattr__(name+'_grad')
                val += p.grad.data
                self.__setattr__(name+'_grad', val)

    def reset_grads(self):
        self.zero_grad()
        for name, p in self.named_parameters():
            if p.grad is not None:
                val = self.__getattr__(name+'_grad')
                val = p.grad.data
                self.__setattr__(name+'_grad', val)

    def synchronize(self):
        for name, p in self.named_parameters():
            val = self.__getattr__(name+'_grad')
            p._grad = Variable(val)
</code></pre>
<p>When a worker have computed a new loss, I use cum_grads:</p>
<pre><code class="lang-auto">ensure_shared_grads(policy, shared_p)
shared_p.cum_grads()
counter.increment()
</code></pre>
<p>And the chief just loops as follow:</p>
<pre><code class="lang-python">def chief(rank, params, traffic_light, counter, shared_p, shared_v, optimizer_p, optimizer_v):
    while True:
        time.sleep(1)
        # workers will wait after last loss computation
        if counter.get() &gt; params.update_treshold:
            shared_p.synchronize()
            shared_v.synchronize()
            optimizer_p.step()
            optimizer_v.step()
            counter.reset()
            shared_p.reset_grads()
            shared_v.reset_grads()
            traffic_light.switch() # workers start new loss computation
</code></pre>

dgriff: <p>It’s looks to be somewhat still asynchronous. One quick test you can do to see if synchronous is that you should no longer have to use a special shared optimizer and should now work with just the regular pytorch optimizers</p>

alexis-jacq: <p>Good point! I don’t longer need the shared optimizer. It still works with a simple one.</p>

alexis-jacq: <p>In fact, my implementation only solves the invertpendulum environment. I am searching for any help, but the paper is not clearly describing the pseudocode (a lot of sub-steps and hyperparameters are not detailed either in DPPO paper and PPO_clip paper). Besides, the only implementation I found is the official <a href="https://github.com/openai/baselines">TF implementation</a> by John Schulman, which I almost rigorously translated into Pytorch, without success.</p>
<p>Unfortunately for me, I can’t even try the TF implementation, since python3 version of mujoco seems incompatible with my machine <img src="//discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:">. It could have at least provided me with some helpful expectation, like what looks like a “working” learning curve.</p>
<p>If any one has some useful documentation or an implementation that works (any language/library) of any close algorithm (TRPO, A2C), please tell me</p>

dgriff: <p>Little confused on somethings I see in <a href="http://train.py" rel="nofollow noopener">train.py</a></p>
<p>What’s the whole updating old model right before you call backward and step about?</p>
<p>Also are you asking for help on converting model to solve other environments or that model is not able to solve other environments?</p>

alexis-jacq: <blockquote>
<p>What’s the whole updating old model right before you call backward and step about?</p>
</blockquote>
<p>If I do it after the backward step or anywhere else, the old and new models will have same weights… I can also update the old model once, it is unclear in the paper if this update must be done once or at every steps.</p>
<blockquote>
<p>Also are you asking for help on converting model to solve other environments or that model is not able to solve other environments?</p>
</blockquote>
<p>According to <a href="https://arxiv.org/abs/1707.06347">the ppo paper</a>, the set of hyperparameters I use in <code>ppo.py</code> should work in every environment with no modification at all. But only if I change both the batch size and the number of steps to 1000, it solves the InvertPendulum (quickly) and the InvertDoublePendulum (much slower than in the paper). With batch size = 64 and number of steps = 2048 as suggested by the paper, It solves nothing.</p>
<p>So I am quite certain there is something wrong somewhere in the training loop, but I can’t figure out what is wrong.</p>

alexis-jacq: <p><a href="https://github.com/ikostrikov/pytorch-a2c-ppo-acktr:">https://github.com/ikostrikov/pytorch-a2c-ppo-acktr:</a> an implementation that seems to work properly!<br>
Now I can go back to this project and check what was wrong with my code.</p>
<p>Thanks and Congrats <a class="mention" href="/u/ilya_kostrikov">@Ilya_Kostrikov</a>!</p>

alexis-jacq: <p>Finally, I found what was wrong, thanks to Oleg Klimov (<a href="https://github.com/openai/baselines/issues/87#issuecomment-363235729">https://github.com/openai/baselines/issues/87#issuecomment-363235729</a>)!</p>

007: <p>Hello, David. This is a really clear and concise implementation of DPPO! Only a question, how do you guarantee the variation in actions between different workers. In my game environment, every worker faces the same initial state, and I use torch.distributions.sample function to sample actions in worker. Unfortunately, I found that the continuous actions sampled by different workers are exactly the same! I use</p>
<p>np.random.seed(params.seed)<br>
torch.manual_seed(params.seed)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False</p>
<p>to guarantee the reproducibility of the algorithm. So I think it might have some influence on the variation between workers. And I wonder how you solve it.</p>