Action produce different results

username: slavavs
cooked: <p>Why does action produce different results?</p>
<pre><code class="lang-auto">import torch as T
import torch.nn.functional as F

f = T.tensor([0.8,0.5,0.4])
f2 = F.softmax(f)
print(f2)
action_probs = T.distributions.Categorical(f2)
print(action_probs)
action = action_probs.sample()
print(action)
</code></pre>

username: albanD
cooked: <p>You sample from a probability distribution here right. So you expect to get different results every time you sample from it.</p>
<p>If you want deterministic behavior, your can fix the random seed for the random generator.</p>

username: slavavs
cooked: <p>I can not understand the logic of T.distributions.Categorical. The neural network gives me three actions. Softmax gives probabilities. In theory, I choose the maximum (argmax). But I can’t understand why:</p>
<pre><code class="lang-auto">action_probs = T.distributions.Categorical(f2)
action = action_probs.sample()
</code></pre>
<p>Can you explain this to me in more detail?))</p>

username: slavavs
cooked: <p>I can’t understand how the connection between the output of the neural network and action = action_probs.sample () is. If the neural network gives the maximum probability for 0, and the action gives 1. What is the point?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The point for the <code>distributions</code> package is to actually sample from distributions, not get the element with max probability.<br>
If you only want the element with max probability, you should use max. If you want to actually sample from the probability that is defined with the weights you have, use distributions.</p>

username: slavavs
cooked: <p>This is an example from the documentation.</p>
<pre><code class="lang-auto">probs = policy_network(state)
m = Categorical(probs)
action = m.sample()
next_state, reward = env.step(action)
loss = -m.log_prob(action) * reward
loss.backward()
</code></pre>
<p>It seems to me more logical to do this:</p>
<pre><code class="lang-auto">probs = policy_network(state)
action  = T.argmax(probs)
m = Categorical(probs)
next_state, reward = env.step(action)
loss = -m.log_prob(action) * reward
loss.backward()
</code></pre>
<p>Explain to me why use action = m.sample ()?</p>
<p>Maybe to better explore the environment?</p>

username: albanD
cooked: <p>You should look into the REINFORCE algorithm for that.<br>
But from what I remember, the gist is that the reward should be the expected reward by following the <code>probs</code> that you computed.<br>
If you take the argmax, you don’t have the expected reward of your computed <code>probs</code> and so you don’t have an unbiased estimate of the gradients.</p>

username: slavavs
cooked: <aside class="quote no-group" data-username="albanD" data-post="7" data-topic="58194">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>But from what I remember, the gist is that the reward should be the expected reward by following the <code>probs</code> that you computed.</p>
</blockquote>
</aside>
<p>Well, logically, if you think about it. The neural network outputs data, I take the maximum value. But if I use action = m.sample (), then I get the action is not clear by what distribution.</p>

username: albanD
cooked: <p><code>m.sample()</code> generate a sample from the distribution given when creating <code>m</code>. Which is <code>probs</code> here.</p>