Creating custom MARL env in torchrl

mcushman: <p>I am using pytorch/torchrl to train a model for playing a variant of poker.  I have implemented the game in an environment following, for example, the example outlines in the pendulum tutorial (<a href="https://pytorch.org/tutorials/advanced/pendulum.html" class="inline-onebox" rel="noopener nofollow ugc">Pendulum: Writing your environment and transforms with TorchRL — PyTorch Tutorials 2.3.0+cu121 documentation</a>).  I would like to make this a multi-agent environment using the MARL API referred to here: <a href="https://pytorch.org/rl/stable/tutorials/multiagent_competitive_ddpg.html" class="inline-onebox" rel="noopener nofollow ugc">Competitive Multi-Agent Reinforcement Learning (DDPG) with TorchRL Tutorial — torchrl 0.4 documentation</a></p>
<p>I cannot find any documentation or tutorials explaining how to implement a multi-agent environment or the details behind the API.  Are there any resources you suggest to understand how the spec should be laid out?</p>

vmoens: <p>Looks like an amazing project!<br>
Do you think you can get enough info here to get started?<br>
<a href="https://pytorch.org/rl/stable/reference/envs.html#multi-agent-environments" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/rl/stable/reference/envs.html#multi-agent-environments</a><br>
Otherwise happy to write more stuff if you can pinpoint what’s blocking you!</p>

mcushman: <p>Thank you for the rapid reply!</p>
<p>I did see that page, and could proceed by analogy to fill out the spec.  It really only gives a single example output, compared with the Pendulum tutorial which gives a fairly complete walk-through of an implementation I find it less useful, but I can try.  I’d also look at some of the code of the existing environments that use the MARL API.</p>
<p>In terms of specific questions, I have two:</p>
<ol>
<li>
<p>As a general principle, is it just a matter of nesting the usual env fields in the specs and output TensorDicts using these examples as a guide?  I was wondering if there are other requirements under the hood that might be missing or obscure from just looking at those outputs.</p>
</li>
<li>
<p>For my particular case, because it’s a competitive turn-based game, the agents do not act in parallel but rather in sequence.  (contrast to the first paragraph in the “Environment” section in the Multiagent Competitive DDPG tutorial).  The environment at any time can return a value of which player’s turn it is to move (or, it’s simply determined by moving in a cycle through the players).  How should this be handled?</p>
</li>
</ol>

monopolize_all: <p>Hi! I was going through the docs you mentioned (<a href="https://pytorch.org/rl/stable/reference/envs.html#multi-agent-environments" class="inline-onebox" rel="noopener nofollow ugc">torchrl.envs package — torchrl 0.5 documentation</a>) but there is no clarification on how we can describe a custom state value(for a centeralized critic in MADDPG) in _step function call. Could you please clarify the same?</p>
<p>More details: I have a custom env, having reset (returns observations, state) and step (returns observations, state, rewards, done). I am trying to wrap it in TorchRL to further test in BenchMARL as a custom task.</p>