How to generate multiple pseudorandom number generators?

kirk86: <p>I’m translating some code from another framework into pytorch and they have a specific use-case of generating some data while training the model.</p>
<p>So, the scenario goes like this:</p>
<pre><code class="lang-auto">rngs_reserve = 10K
random.seed(seed)
np.random.seed(seed)
rng_seq = PRNGSequence(seed)
rng_seq.reserve(rngs_reserve) # reserve 10000 PRNG keys

# training loop
for step in steps: # run for 10K steps
    data = sample_batch(next(rng_seq), ....) # this uses random generation e.g. torch.randint(generator=rng)
    output = model(data)
    .
    .
    .
</code></pre>
<p>As you can notice the <code>rng_seq</code> is an iterator which every time returns a new pseudorandom key which is then passed on to the function generating random numbers, e.g., <code>torch.randint(generator=rng)</code>.</p>
<p>So my attempt at converting this into pytorch is the following:</p>
<pre><code class="lang-auto">random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)

# training loop
for step in steps: # run for 10K steps
    rng = torch.Generator()
    data = sample_batch(rng, ....) # this uses random generation e.g. torch.randint(generator=rng)
    output = model(data)
    .
    .
    .
</code></pre>
<p>Questions:<br>
Is this approach faithfully following the original approach?<br>
The goal is to generate diverse data samples during the training so that we minimize overlap between samples generated on the fly, is there a better way to achieve this?</p>

KFrank: <p>Hi Kirk!</p>
<p>The short story is that you really don’t want to use multiple pseudorandom<br>
number generators.  (Doing so can perversely make your numbers less<br>
pseudorandom.)  For most use cases, it will likely be best to use pytorch’s<br>
global random number generator as it is automatically called by pytorch’s<br>
various random functions.</p>
<p>(Distributed processing is a case where it makes sense to have multiple<br>
random number generators running on multiple processors.)</p>
<aside class="quote no-group" data-username="kirk86" data-post="1" data-topic="196050" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<pre><code class="lang-auto"># training loop
for step in steps: # run for 10K steps
    rng = torch.Generator()
    data = sample_batch(rng, ....) # this uses random generation e.g. torch.randint(generator=rng)
</code></pre>
</blockquote>
</aside>
<p>To answer your specific question:  Yes, you can do something like this, but<br>
you need to initialize each instance of <code>rng</code>.  You can call <code>rng.seed() </code> to<br>
initialize each <code>rng</code> instance to some sort of “random” seed, or you can call<br>
<code>rng.manual_seed (seed)</code> to initialize <code>rng</code> to your specific choice for <code>seed</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>The goal is to generate diverse data samples during the training so that we minimize overlap between samples generated on the fly,</p>
</blockquote>
</aside>
<p>Yes, this is the issue.  Unless you use a nuanced scheme for initializing your<br>
individual <code>rng</code>s, you do risk getting “overlap between samples generated on<br>
the fly,”</p>
<aside class="quote no-group">
<blockquote>
<p>is there a better way to achieve this?</p>
</blockquote>
</aside>
<p>Yes, don’t use multiple <code>rng</code>s (that you would have to initialize very carefully).<br>
Just use pytorch’s global random number generator that (if well designed)<br>
will not have “overlap” between subsequent sets of samples.</p>
<p>Best.</p>
<p>K. Frank</p>

kirk86: <p>Hi KFrank, thanks for your reply.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="196050">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Yes, don’t use multiple <code>rng</code>s (that you would have to initialize very carefully).<br>
Just use pytorch’s global random number generator that (if well designed)<br>
will not have “overlap” between subsequent sets of samples.</p>
</blockquote>
</aside>
<p>So, when you mention <code>pytorch global random number generator</code>  are you referring to <code>torch.manual_seed()</code> or something else?  Could you please clarify with a MWE?</p>
<p>If we indeed end up using pytorch’s global rng wouldn’t that create samples with overlaps if for instance we sample lots of numbers? e.g., millions?</p>
<p>Another way I was thinking about doing this was using local number generators. For instance, in the code set you seeds for reproducibility and then in the train loop use different seeds for <code>rng</code>.</p>
<p>MWE of <code>main.py</code></p>
<pre><code class="lang-auto"># set the seed for different random functions
os.environ['PYTHONHASHSEED'] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)


# training loop
rng = torch.Generator() # Is this better to be one instane or generate multiple inside the for loop?
for step in steps: # run for 10K or 10M steps
    rng.manual_seed(step)
    data = sample_batch(rng, ....) # this uses random generation e.g. torch.randint(generator=rng)
</code></pre>
<p>The <code>seed != step</code>, so <code>seed</code> and <code>step</code> are always different to avoid overlap.</p>

KFrank: <p>Hi Kirk!</p>
<p>Again, the short story is that people have invested a lot of effort into<br>
implementing really good random number generators.  Use what they’ve<br>
done, as provided to you out-of-the-box by pytorch.</p>
<aside class="quote no-group" data-username="kirk86" data-post="3" data-topic="196050" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>So, when you mention <code>pytorch global random number generator</code>  are you referring to <code>torch.manual_seed()</code> or something else?</p>
</blockquote>
</aside>
<p>Pytorch’s official documentation is mostly silent about the details of its<br>
random number generators.</p>
<p>What appears to be happening is that when you load pytorch (or maybe<br>
when you first access some random numbers), pytorch instantiates a<br>
<code>Generator</code>, initialized with some sort of a “random” seed, and makes it<br>
available “globally.”</p>
<p>That is, all of pytorch’s functions that use random numbers call this<br>
<code>Generator</code> to get their random numbers (unless you tell them to do<br>
otherwise by passing in a <code>generator</code> argument).</p>
<p>So:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn (8)
y = torch.randperm (3)
z = torch.nn.Linear (2, 5)
</code></pre>
<p>all consume random numbers from the global <code>Generator</code> (in the order<br>
they are called).</p>
<p>When you call <code>torch.manual_seed (some_seed)</code>, you are reinitializing<br>
this global <code>Generator</code> with <code>some_seed</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>If we indeed end up using pytorch’s global rng wouldn’t that create samples with overlaps if for instance we sample lots of numbers? e.g., millions?</p>
</blockquote>
</aside>
<p>As far as I can tell, pytorch uses, on the cpu*, a Mersenne twister algorithm<br>
for its pseudorandom number generator, most likely the so-called “MT19937.”<br>
This generator has a period of <code>2**19937 - 1</code>.  So the sequence it produces<br>
contains <em>lots and lots</em> of millions of sample,  To get overlaps, you would<br>
have to sample more than <code>2**19937 - 1</code> numbers.  (And if you did, you<br>
would be in the soup with virtually all random number generators.  Plus, your<br>
application would take “a while” to run.)</p>
<aside class="quote no-group">
<blockquote>
<p>Another way I was thinking about doing this was using local number generators.<br>
…<br>
The <code>seed != step</code>, so <code>seed</code> and <code>step</code> are always different to avoid overlap.</p>
</blockquote>
</aside>
<p>If you do this, you will <em>dramatically</em> increase the probability of overlaps.<br>
(In practice, you will probably be fine, but why go to the extra work just<br>
to do something that is worse, even if it still works in practice?)</p>
<p>Note that using multiple copies of <code>Generator</code> doesn’t create more random<br>
numbers – it just gives different chunks of the length-<code>2**19937 - 1</code><br>
sequence generated by a single copy of <code>Generator</code>.  That sequence<br>
won’t overlap until you’ve exhausted its full length.  Your multiple copies<br>
have no guarantee that they won’t overlap (but it’s unlikely that they will<br>
unless you have lots of individual <code>Generator</code>s or extract lots of numbers<br>
from them).</p>
<p>*)  On the gpu, I believe that pytorch uses the so-called “philox” algorithm.<br>
A philox stream should have a period of at least <code>2**128</code> – still pretty long,<br>
although <em>much</em> shorter than that of MT19937 – but can produce at least<br>
<code>2**64</code> independent (“non-overlapping”) streams, making it suitable for<br>
running multiple generators on multiple gpus.</p>
<p>Best.</p>
<p>K. Frank</p>