Calling torch.distributions.categorical.Categorical multiple times can affect the final result

wqingze: <p>Calling <code>torch.distributions.categorical.Categorical</code> multiple times can affect the final result.</p>
<p>In the PPO (Proximal Policy Optimization) algorithm, the action function is defined as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">def choose_action(self, s, evaluate=False):
    with torch.no_grad():
        s = torch.tensor(s, dtype=torch.float).unsqueeze(0)
        logits = self.ac.actor(s.cuda())
        if evaluate:
            a = torch.argmax(logits)
            return a.item(), None
        else:
            dist = Categorical(logits=logits)
            a = dist.sample()
            a_logprob = dist.log_prob(a)
            return a.item(), a_logprob.item()
</code></pre>
<p>In practice, it is observed that different calls to <code>Categorical</code> result in different outcomes. For example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.distributions import Categorical
m = Categorical(torch.tensor([0.25, 0.25, 0.25, 0.25]))
for _ in range(10000):
    m.sample()

ppo = PPO()
env = Env(config, task='test')
evaluate(ppo, env)

# Output result: reward = 10
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">ppo = PPO()
env = Env(config, task='test')
evaluate(ppo, env)

# Output result: reward = 100
</code></pre>

KFrank: <p>Hi Qingze!</p>
<aside class="quote no-group" data-username="wqingze" data-post="1" data-topic="200238" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqingze/48/46172_2.png" class="avatar"> wqingze:</div>
<blockquote>
<p>In practice, it is observed that different calls to <code>Categorical</code> result in different outcomes.</p>
</blockquote>
</aside>
<p>I believe that you’re asking why calls to <code>Categorical.sample()</code> (and<br>
thing computed from them) aren’t reproducible.</p>
<p>This is to be expected as <code>.sample()</code> returns pseudorandom samples<br>
from the <code>Categorical</code> distribution you instantiated.  These samples<br>
are supposed to be (randomly) different from one another.  If you want<br>
a series of <code>.sample()</code>s to be reproducible, you may reset the underlying<br>
random-number generator with <code>torch.manual_seed()</code>.  Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.2.2'
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt; m = torch.distributions.Categorical(torch.tensor([0.25, 0.25, 0.25, 0.25]))
&gt;&gt;&gt; m.sample(), m.sample()         # two samples
(tensor(2), tensor(1))
&gt;&gt;&gt; m.sample(), m.sample()         # two more samples -- different result
(tensor(3), tensor(2))
&gt;&gt;&gt; _ = torch.manual_seed (2024)   # reset random number generator
&gt;&gt;&gt; m.sample(), m.sample()         # same two samples as at the beginning
(tensor(2), tensor(1))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

wqingze: <p>I may have caused some confusion for you.</p>
<p>choose_action() is a member funcion of PPO class. I delare a ppo object, and call its choose_action() in evaluate() function<br>
if call .sample() before evaluate() many times, ppo give a better reward</p>
<p>I know that .sample() is not repoducible, but the problem is that .sample() may have memory, call .sample() many times before call choose_action(), will give me a more better reward.</p>
<p>I have set seed by torch.manual_seed(seed)</p>

KFrank: <p>Hi Qingze!</p>
<aside class="quote no-group" data-username="wqingze" data-post="3" data-topic="200238" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqingze/48/46172_2.png" class="avatar"> wqingze:</div>
<blockquote>
<p>if call .sample() before evaluate() many times, ppo give a better reward</p>
</blockquote>
</aside>
<p>This is just* a (pseudo) statistical happenstance cause by the different,<br>
but statistically-equivalent, (pseudo) random samples you get after<br>
“advancing the random-number-generator state” by calling <code>.sample()</code><br>
a bunch of times (and discarding the results) before performing your<br>
actual computation.</p>
<aside class="quote no-group">
<blockquote>
<p>I know that .sample() is not repoducible, but the problem is that .sample() may have memory</p>
</blockquote>
</aside>
<p><code>.sample()</code> (or, more precisely, an instance of <code>Categorical</code>) does not<br>
have memory.  (Pytorch’s random-number generator does have state.<br>
That state changes the specific <em>values</em> of the samples generated, but<br>
not their statistical characteristics.)</p>
<p>Try this experiment, say ten to one hundred times.  Start a new python<br>
session and <code>import pytorch</code>.  This initializes pytorch’s random-number<br>
generator to a new, “random,” state.  Either compute your reward or<br>
call <code>.sample()</code> a bunch of times and then compute your reward.  You<br>
will find that there is no <em>statistical</em> difference between the rewards you<br>
get with and without calling <code>.sample()</code> a bunch of times.</p>
<aside class="quote no-group">
<blockquote>
<p>I have set seed by torch.manual_seed(seed)</p>
</blockquote>
</aside>
<p>*)  There is some lore that the quality of the pseudorandom numbers<br>
initially produced by pytorch’s random-number generator is reduced if<br>
<code>seed</code> has only low-order bits.  So, if you want to be extra-careful about<br>
this (I don’t bother.), you should use a <code>seed</code> that comes from basically<br>
the whole range of a 64-bit integer.</p>
<p>Best.</p>
<p>K. Frank.</p>