Multidmensional Actions

username: ar795
cooked: <p>Hello there,<br>
Please,How can we apply Reinforce or any Policy gradient algorithm when the actions space is multidimensional, let’s say that for each state the action is a vector a= [a_1,a_2,a_3] where a_i are discrete ?<br>
In this case the output of  the policy network must model a joint probability distribution<br>
Thanks</p>

username: iffiX
cooked: <p>You need the reparameterization trick:</p>
<p>if each dimension of your action vector is of the same meaning (same number and meaning of choices), consider about <code>torch.multinomial</code></p>
<p>if not and each dimension are independent from each other, then create multiple outputs in your network and direct them to multiple categorical distribution, and draw each dimension of your sample from these distributions.</p>
<p>If they are correlated, then you need to build your own model of distribution.</p>
<p>Another option is to map all combinations: a_1 x a_2 x a_3 to a single categorical distribution.</p>

username: ar795
cooked: <p>Thanks for your reply ,<br>
Yes,  the action vector components are all of the same meaning :<br>
Let’s say that the output of my policy network is a 2D tensor of shape (m,n) (sum of each row equals one)<br>
and I want to generate according to this probability matrix an action vector of size m a = [a_1,…,a_m]<br>
In this case the multinomial distributions is the suitable one ?</p>

username: iffiX
cooked: <p>yes，you should use the multinomial distribution.</p>

username: Ebrahim.Pichka
cooked: <p>It’s been a while since the question has been asked, yet since I recently faced the same issue I’ll add another answer just for reference and also others’ feedback on its correctness.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36365b0f0802c9a298139f05567d41f3f7c9253f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/36365b0f0802c9a298139f05567d41f3f7c9253f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36365b0f0802c9a298139f05567d41f3f7c9253f_2_345x194.jpeg" alt="image" data-base62-sha1="7JAeSmT5pulJGWYDjkvUH5puysL" width="345" height="194" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36365b0f0802c9a298139f05567d41f3f7c9253f_2_345x194.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36365b0f0802c9a298139f05567d41f3f7c9253f_2_517x291.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36365b0f0802c9a298139f05567d41f3f7c9253f_2_690x388.jpeg 2x" data-dominant-color="B68DEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 67.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(Image from the <a href="https://huggingface.co/learn/deep-rl-course/unit4/policy-gradient" rel="noopener nofollow ugc">HuggingFace course</a>)</p>
<p>Following the PG theorem, we need to decompose the joint probability to the product of prob of each action. assuming the actions are independent, we get.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9bd3023f3a09f5d80a52db61725b30fbe3215b1.png" alt="image" data-base62-sha1="zDidRLZB5tSaXCA6zDrCIKj0iJj" width="349" height="50"></p>
<p>Thus, the log-probs just get added together.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e093fdb7216d70a7dd12f1a0b13ef0ebb1fd6d3a.png" alt="image" data-base62-sha1="w2I1pDlh5pwf9UGtqZU6U7seuKK" width="497" height="50"></p>
<p>in Pytorch,you’d need to consider 3 separate output heads for the Actor, one for each action (or 3 separate networks, which will be inefficient in learning). It’ll be as</p>
<pre data-code-wrap="py"><code class="lang-py">class ActorNet(nn.Module):
    def __init__(..., a1_dim, a2_dim, a3_dim, ...):
        # ... define arch
        self.a1_head = nn.linear(embed_dim, a1_dim)
        self.a2_head = nn.linear(embed_dim, a2_dim)
        self.a3_head = nn.linear(embed_dim, a3_dim)

    def forward(obs):
        embed = self.arch(obs)
        act1_logits = self.a1_head(embed )
        act2_logits = self.a2_head(embed )
        act3_logits = self.a3_head(embed )
        return act1_logits, act2_logits, act3_logits

actor = ActorNet(..., 5, 4, 8, ...)  # would work with different action dims
</code></pre>
<p>Then while collecting rollouts:</p>
<pre data-code-wrap="py"><code class="lang-py">from torch.distributions import Categorical

def get_action(obs):
    act1_logits, act2_logits, act3_logits  = actor(obs)

    dist_1 = Categorical(logits=act1_logits)
    dist_2 = Categorical(logits=act2_logits)
    dist_3 = Categorical(logits=act3_logits)

    act_1 = dist_1.sample()
    act_2 = dist_2.sample()
    act_3 = dist_3.sample()

    log_prob = dist_1.log_prob(act_1) + dist_2.log_prob(act_2) + dist_3.log_prob(act_3)
    return (act_1, act_2, act_3) , log_prob 
</code></pre>
<hr>
<p>Now that we have one <code>log_prob</code> and all 3 actions, the rest of the PG algorithm follows the same it normal case.</p>
<p><strong>Note:</strong> When estimating the advantage in Actor Critic models, if you use <em><strong>Q</strong></em> as the critic, you will need <code>a1_dim * a2_dim * a3_dim</code> outputs since each action combination is a unique overall action, which gets combinatorically large! Using <em><strong>V</strong></em> would be more efficient.</p>