Why does pytorch have different devices of the random number generator, if no default one is given?

potatolige: <p>Hi,</p>
<p>I am facing an issue caused by the random number generator.</p>
<p>As my code works in most cases on GPU, and I am too lazy to manually set the device when I initialize a tensor through <code>torch.Tensor(...)</code>,</p>
<p>I set the default tensor’s type to cuda.FloadTensor by <code>torch.set_default_tensor_type('torch.FloatTensor')</code></p>
<p>Then I found all the random number generation operation, such as <code>torch.randperm(3)</code>, will return me a result as a GPU tensor, even if I did not specify the random number generator’s devcie to cuda. Very Cool!</p>
<p>However, since my dataset is not very big, I decide to save it in GPU memory, and use a dataloader to load minibatches. Then I found if I do not specify a cuda generator <code>generator=torch.Generator(device="cuda")</code> when I initialize the dataloader, it will use the cpu as the default device and thus raise an exception when I yield a minibatch from the loader:  <code>RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'</code>.</p>
<p>So why is this the case, i.e. not taking cuda as the default generator’s device type? Why is the behaviour here different to, e.g., <code>torch.randperm(3)</code> case?</p>
<p>Besides, since I want to control the randomness and reproduce my result, I always fix the randomness seed in each run, e.g. 1234, and according to the <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">Controlling sources of randomness</a>, calling <code>torch.manual_seed(0)</code> will set the seed to 0 for generators of both cpu’s and gpu’s breifly in one line. But this seed will not influence the gpu generator of the dataloader. I need to formally initialize a gpu generator, set the seed manually and then initilize the dataloader. Otherwise, I cannot make sure that the dataloader’s seed is the same to the other operations like <code>torch.randperm(3)</code>.</p>
<p>Since it is a solvable issue, I did not put it in github’s issue. But it breaks the beauty of setting randomness globally and brings exception of the official tutorial. So I would like to hear any suggestion from yours.</p>
<p>Best,<br>
Bruce</p>

srishti-git1110: <p>Hi,<br>
I was working on something related and came across this post - nice one!</p>
<p>Could you please elaborate what exactly you meant by saving the data in the GPU memory?<br>
Did you do that in the <code>__init__()</code> or the <code>__getitem()__</code> method of the <code>Dataset</code> class or did you mean something else?</p>

ptrblck: <aside class="quote no-group" data-username="srishti-git1110" data-post="2" data-topic="147343">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srishti-git1110/48/52515_2.png" class="avatar"> srishti-git1110:</div>
<blockquote>
<p>I was working on something related and came across this post - nice one!</p>
</blockquote>
</aside>
<p>From my experience: setting the default tensor type to a CUDATensor was never properly working for <em>all</em> methods and some operations were breaking, just as a heads-up in case you want to use the same approach.</p>

potatolige: <p>Hi, I meant I just save all my data in a big tensor with device=“cuda”, I found it is faster than save it in cpu tensor and get a batch out of it and then change the device to “cpu”.</p>
<p>I think in the lastest version of pytorch 1.12, they have changed the usage of dataloader. Now it is not possible to give a generator to the init function of dataloader anymore. So I guess the bug has been fixed somehow.</p>