Centralized learning-decentralized execution clarification (engineering perspective on PPO algo)

username: Kimonili
cooked: <p>Hi everyone,</p>
<p>I can understand the theoritical concept of the centralized learning-decentralized execution approach, but I am quite confused about the coding-engineering changes to be done in the update of the networks in the PPO algo.</p>
<p>I think that the actor network (I have seperate networks) will use each agent’s actor loss to update the network, but how the critcs are updated? Should I calculate the cummulative critic loss (from all the agents) and backpropagate it in every single critic network?</p>

username: iffiX
cooked: <p>Is it in a multi-agent scenario? Do you want to create a multi-agent PPO framework similar to MADDPG?</p>

username: Kimonili
cooked: <p>Yes, that is correct. I found this <a href="https://www.mdpi.com/1424-8220/20/16/4546/html" rel="nofollow noopener">paper</a> that created a multiagent version of the PPO algorithm with a centralized approach. The problem is that they have not published the code yet.</p>

username: iffiX
cooked: <p>Well, then its very simple, critic <code>V(x)_i</code> of agent i will collect the current observed state of agent i and the so called “partial state” of other agents j, then predict a value. So the code structure of the training part would be parallelly sample an action from all agent policies <code>P(x)_m, m \in {0, n}</code> where n is the number of agents, then you can exchange the state information between all agents, then parallelly use critics to evaluate values, then parallely update critics, and finally parallely update actors using values produced by critics.</p>
<p>The simplest (with medium performance) way to implement this is using a threadpool, eg: the one from <code>joblib</code> or <code>multiprocessing</code> module, if your model is small. A more complex way would be using <code>torch.jit.fork_</code>, which is even better than using the threadpool because it completely avoids the GIL problem (about 50% faster than threadpool). <code>ProcessPool</code> is only recommended if your model is on aa GPU and very large (like using a resnet or something of similar complexity and parameter size, there are many caveats in using a ProcessPool and I don’t recommend it.</p>

username: Kimonili
cooked: <p>So in general, there will be N=number_of_agents critic and actor networks. The centralized learning appears in the fact that each critic network recieves state information from the other agents and then updates its weights?</p>
<aside class="quote no-group" data-username="iffiX" data-post="4" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>sample an action from all agent policies <code>P(x)_m, m \in {0, n}</code> where n is the number of agents</p>
</blockquote>
</aside>
<p>So here all the agents make an action simultaneously.</p>
<aside class="quote no-group" data-username="iffiX" data-post="4" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>then you can exchange the state information between all agents</p>
</blockquote>
</aside>
<p>Given the action each agent made in the previous step, they transision to a new state. The state of each agent will be shared to all the agents. So basically, every agent will be aware of the position of every agent.</p>
<aside class="quote no-group" data-username="iffiX" data-post="4" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>then parallelly use critics to evaluate values</p>
</blockquote>
</aside>
<p>Here the critics are evaluating its agent’s actions but being aware of the new dynamics of the environments? (after getting informed of the relative position of every agent)</p>
<aside class="quote no-group" data-username="iffiX" data-post="4" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>then parallely update critics, and finally parallely update actors using values produced by critics.</p>
</blockquote>
</aside>
<p>This step is related to the previous one so the question remains the same.</p>
<aside class="quote no-group" data-username="iffiX" data-post="4" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>The simplest (with medium performance) way to implement this is using a threadpool, eg: the one from <code>joblib</code> or <code>multiprocessing</code> module, if your model is small. A more complex way would be using <code>torch.jit.fork_</code> , which is even better than using the threadpool because it completely avoids the GIL problem (about 50% faster than threadpool). <code>ProcessPool</code> is only recommended if your model is on aa GPU and very large (like using a resnet or something of similar complexity and parameter size, there are many caveats in using a ProcessPool and I don’t recommend it.</p>
</blockquote>
</aside>
<p>I believe then that the best choice is <code>torch.jit.fork_</code> , which I understood that can run in CPU as well.</p>
<p>I am really sorry for the questions, I just want to be sure I understand this correctly! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Thank you so much!</p>

username: iffiX
cooked: <aside class="quote no-group" data-username="Kimonili" data-post="5" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kimonili/48/38707_2.png" class="avatar"> Kimonili:</div>
<blockquote>
<p>So in general, there will be N=number_of_agents critic and actor networks. The centralized learning appears in the fact that each critic network recieves state information from the other agents and then updates its weights?</p>
</blockquote>
</aside>
<p>Yes</p>
<aside class="quote no-group" data-username="Kimonili" data-post="5" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kimonili/48/38707_2.png" class="avatar"> Kimonili:</div>
<blockquote>
<p>So here all the agents make an action simultaneously.</p>
</blockquote>
</aside>
<p>Yes</p>
<aside class="quote no-group" data-username="Kimonili" data-post="5" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kimonili/48/38707_2.png" class="avatar"> Kimonili:</div>
<blockquote>
<p>Given the action each agent made in the previous step, they transision to a new state. The state of each agent will be shared to all the agents. So basically, every agent will be aware of the position of every agent.</p>
</blockquote>
</aside>
<p>Yes</p>
<aside class="quote no-group" data-username="Kimonili" data-post="5" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kimonili/48/38707_2.png" class="avatar"> Kimonili:</div>
<blockquote>
<p>Here the critics are evaluating its agent’s actions but being aware of the new dynamics of the environments? (after getting informed of the relative position of every agent)</p>
</blockquote>
</aside>
<p>Well, PPO critic do not evaluate actions of agents, just state of the current agent, and partial states from other agents, since the critic represents value function <code>V(s)</code> and not Q value function <code>Q(s, a)</code>.</p>
<aside class="quote no-group" data-username="Kimonili" data-post="5" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kimonili/48/38707_2.png" class="avatar"> Kimonili:</div>
<blockquote>
<p>Here the critics are evaluating its agent’s actions but being aware of the new dynamics of the environments? (after getting informed of the relative position of every agent)</p>
</blockquote>
</aside>
<p>Yes</p>
<p>A reference to my MADDPG implementation, includes three variants of parallel mechanism mentioned above, if you would like to take a look:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/iffiX/machin/blob/master/machin/frame/algorithms/maddpg.py">
  <header class="source">

      <a href="https://github.com/iffiX/machin/blob/master/machin/frame/algorithms/maddpg.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/iffiX/machin/blob/master/machin/frame/algorithms/maddpg.py" target="_blank" rel="noopener nofollow ugc">iffiX/machin/blob/master/machin/frame/algorithms/maddpg.py</a></h4>


      <pre><code class="lang-py">import copy
import inspect
import itertools
from random import choice, randint
from .utils import determine_device
from machin.utils.visualize import make_dot
from machin.utils.logging import default_logger
from machin.model.nets.base import static_module_wrapper
from machin.parallel.pool import P2PPool, ThreadPool
from machin.frame.transition import Scalar

# pylint: disable=wildcard-import, unused-wildcard-import
from .ddpg import *


class SHMBuffer(Buffer):
    def make_tensor_from_batch(
        self,
        batch: List[Union[Scalar, t.Tensor]],
        device: Union[str, t.device],
</code></pre>



  This file has been truncated. <a href="https://github.com/iffiX/machin/blob/master/machin/frame/algorithms/maddpg.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Kimonili
cooked: <p><a class="mention" href="/u/iffix">@iffiX</a> Thank you very much! I will take a look right away! You were very informative!</p>

username: Kimonili
cooked: <p>Hi <a class="mention" href="/u/iffix">@iffiX</a>,</p>
<p>I used the multiprocessing module in order to parallelize the experiment in the multiagent particle environment. I saw that you used the same env as well. What happens in my code is that when I apply the step function (the one from the environment’s module) I get this error:</p>
<pre><code class="lang-auto">_pickle.PicklingError: Can't pickle &lt;class '.Scenario'&gt;: import of module '' failed
</code></pre>
<p>Apparently this Scenario class can’t be pickled. Are you aware of a wrapper that avoids this in some way?</p>

username: iffiX
cooked: <p>I would recommend you send a command to create the environment inside the subprocess directly, instead of pickling it, there are many problems related to pickling gym-based environments and are way too painful to deal with.</p>
<p>I made a wrapper for parallelizing openai gym execution in subprocesses at:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/iffiX/machin/blob/master/machin/env/wrappers/openai_gym.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/iffiX/machin/blob/master/machin/env/wrappers/openai_gym.py" target="_blank" rel="nofollow noopener">iffiX/machin/blob/master/machin/env/wrappers/openai_gym.py</a></h4>
<pre><code class="lang-py">from itertools import repeat
from typing import Tuple, Callable
from threading import Lock
from multiprocessing import get_context
import gym
import numpy as np

from machin.parallel.exception import ExceptionWithTraceback
from machin.parallel.queue import SimpleQueue
from machin.parallel.pickle import dumps, loads

from .base import *
from ..utils.openai_gym import disable_view_window


class GymTerminationError(Exception):
    def __init__(self):
        super(GymTerminationError, self).__init__(
            "One or several environments have terminated, "
            "reset before continuing."
</code></pre>

  This file has been truncated. <a href="https://github.com/iffiX/machin/blob/master/machin/env/wrappers/openai_gym.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
It might will also work for the particle env, but I have not tested it.

username: Kimonili
cooked: <p>Thank you <a class="mention" href="/u/iffix">@iffiX</a>, again I am gratefull for your feedbacks. Just one last question, how is the state of each agent shared to every critic network of all agents?</p>
<p>What I do not understand is this: If we had decentralized learning, each critic network of every <code>agent i</code> would recieve as input just one vector, the state vector of the <code>agent i</code>. Now that the states are shared, how should I add the state vectors of the other agents into the <code>agent's i</code> critic network?</p>
<p>Should it be a list of tensors, where each element of the list is the state tensor of each agent in the environment?</p>
<p>Finally, I was thinking that the state of <code>agent i</code> must greater affect the policy of the <code>agent i</code> compared to the effect of the state of the <code>agent j</code> or <code>agent z</code> to the policy of <code>agent i</code>. If this is true, the network’s weights will figure this out and adjust the weights in order to make more significant the state vector of <code>agent i</code>?</p>
<p>Thank you in advance!</p>

username: iffiX
cooked: <p>In the centralized training, decentralized acting scenario, yes, critics will observe state vectors of other agents, and it is usually done by collecting states of all agents and concatenate them into a fixed dimension state tensor and feed into the critic network. You can of course permute the order of states if your agents are homogenous, this <strong>may</strong> increase network robustness.</p>
<p>It is really unknown whether your network will take <code>agent i</code> more seriously than other agents, however, you may explicitly apply the attention mechanism, which is usually used in NLP areas, to sort of peeking into the thought of your critic network.</p>

username: Kimonili
cooked: <p>Really interesting conversation.</p>
<p>Another topic is whether its better to have one critic network per agent or one critic network for each cluster of homogenous agents, as in the centralized training scenario the input vector will be the same for every critic. Some claim that this might have better performance.<br>
However, I suppose that the attention mechanism scenario is can only be applied for multiple critic networks.</p>
<blockquote>
<p>You can of course permute the order of states if your agents are homogenous, this  <strong>may</strong>  increase network robustness.</p>
</blockquote>
<p>What I did is concatenating the states and feed them to every critic network with the same order. I could instead try to always feed the critic of <code>agent i</code>, with the state of <code>agent i</code> being in the start of the concatenated states.</p>
<p>Thank you for your detailed answer <a class="mention" href="/u/iffix">@iffiX</a> !</p>

username: iffiX
cooked: <p>I have tried:</p>
<ol>
<li>permute order randomly</li>
<li>make <code>agent i</code> first</li>
<li>order states from 1 to n<br>
In my undergraduate thesis, there is no significant difference.</li>
</ol>
<p>A visualization of attention weight, using method 3:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf72b5667277a4672a49f280e99db1f5176d07cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf72b5667277a4672a49f280e99db1f5176d07cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf72b5667277a4672a49f280e99db1f5176d07cc_2_690x294.png" alt="image" data-base62-sha1="rjCXrXnnQuo5SgIU5G9BxZnLUZe" width="690" height="294" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf72b5667277a4672a49f280e99db1f5176d07cc_2_690x294.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf72b5667277a4672a49f280e99db1f5176d07cc_2_1035x441.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf72b5667277a4672a49f280e99db1f5176d07cc.png 2x" data-dominant-color="DBE5EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1063×454 33.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For single/multiple critics in the homogenous scenario:<br>
I have tried:</p>
<ol>
<li>one central critic</li>
<li>one critic for every agent</li>
<li>one critic for every agent, but average their parameters every few episodes. ( by mean() their state dict and broadcast new mean parameters to all critics)<br>
1 performs best and 3 performs the worst, I assume that averaging has damaged the information collected by critic.</li>
</ol>
<p>You may try these methods yourself as I have only test my solution with one multi-agent environment modified from the bipedalwalker-v2, contact me if you would like to take a look at the modified implementation:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e1dec25d12aa7d4de026d591289d398d67baf39.png" alt="image" data-base62-sha1="b93lMdoiUTg53949WCcYLPeGdJL" width="600" height="400"></p>

username: Kimonili
cooked: <p><a class="mention" href="/u/iffix">@iffiX</a> impressive experimentation, congratulations!!</p>
<aside class="quote no-group" data-username="iffiX" data-post="13" data-topic="94195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>You may try these methods yourself as I have only test my solution with one multi-agent environment modified from the bipedalwalker-v2, contact me if you would like to take a look at the modified implementation:</p>
</blockquote>
</aside>
<p>I will, thank you very much for the informaiton! These are very interesting topics.</p>