Generate equal random variable even if dtype is different

tsp_t: <p>Seeing that we get the same bernoulli_ after uniform_, I think the seed is managed regardless of whether we are operating on float32 or float16.<br>
By the way, unlike the result I got from the code, I want to make the normal_() obtained from the tensor of float16 the same as the value obtained from the normal_() generated from float32 (without type casting).</p>
<p>Any ideas?</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(42)
print(torch.empty(10, dtype=torch.float32).uniform_())
print(torch.empty(10, dtype=torch.bool).bernoulli_())
torch.manual_seed(42)
print(torch.empty(10, dtype=torch.float16).uniform_())
print(torch.empty(10, dtype=torch.bool).bernoulli_())
</code></pre>
<pre><code class="lang-auto">tensor([0.8823, 0.9150, 0.3829, 0.9593, 0.3904, 0.6009, 0.2566, 0.7936, 0.9408,
        0.1332])
tensor([ True, False, False, False, False,  True, False, False, False, False])
tensor([0.5498, 0.7124, 0.4199, 0.6318, 0.5518, 0.5347, 0.8418, 0.5098, 0.7998,
        0.0591], dtype=torch.float16)
tensor([ True, False, False, False, False,  True, False, False, False, False])
</code></pre>

ptrblck: <aside class="quote no-group" data-username="tsp_t" data-post="1" data-topic="161193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tsp_t/48/47302_2.png" class="avatar"> tsp_t:</div>
<blockquote>
<p>By the way, unlike the result I got from the code, I want to make the normal_() obtained from the tensor of float16 the same as the value obtained from the normal_() generated from float32 (without type casting).</p>
</blockquote>
</aside>
<p>I don’t know how this should work, since you cannot represent all <code>float32</code> values in <code>float16</code>. Sampling <code>normal_</code> values in <code>float16</code> thus has to stick to valid <code>float16</code> values.</p>

tsp_t: <p>Assuming R16/R32 is a tensor with normal_ operation performed on a tensor of float16/float32 respectively, I expected the following to be correct.</p>
<pre><code class="lang-auto">diff = (R16 + (R16.float() -  R32).half())-R16
assert diff.max().item() == 0
</code></pre>
<p>I have not tried it on other gpu, but I have confirmed that the condition is satisfied in the code executed on MX250@2GB. However, the CPU behaves differently.</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(42)
R32 = torch.empty(10, dtype=torch.float32, device="cuda").uniform_()
torch.manual_seed(42)
R16 = torch.empty(10, dtype=torch.float16, device="cuda").uniform_()

diff_fp16 = (R16 + (R16.float() -  R32).half())-R16
assert diff_fp16.max().item() == 0, diff_fp16.max().item()
</code></pre>

ptrblck: <p>In your current code snippet it seems you are expecting that:</p>
<pre data-code-wrap="python"><code class="lang-python">fl32_eps  R16.float() -  R32
fp16_eps  = fp32_eps.half()
fp16_diff = R16 - fp16_eps
</code></pre>
<p>underflows and returns <code>R16</code> again since <code>diff</code> is expected to be a perfect <code>0.</code>.<br>
I don’t understand how this is related to:</p>
<blockquote>
<p>I want to make the normal_() obtained from the tensor of float16 the same as the value obtained from the normal_() generated from float32</p>
</blockquote>

tsp_t: <p>I want the following two tensors to have the same value.</p>
<ol>
<li>Initialize the tensor of float32 to normal_ (R32) and then typecast to float16 (R32to16)</li>
<li>Initialize the tensor of float16 to normal_ (R16)</li>
</ol>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="161193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>since you cannot represent all <code>float32</code> values in <code>float16</code>.</p>
</blockquote>
</aside>
<p>Reflecting the point that it is impossible to say that they have the same value here, I wanted to make sure that both tensors have the same value when expressed as a float16. (R16 == R32to16)</p>
<aside class="quote no-group" data-username="tsp_t" data-post="3" data-topic="161193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tsp_t/48/47302_2.png" class="avatar"> tsp_t:</div>
<blockquote>
<pre><code class="lang-auto">diff_fp16 = (R16 + (R16.float() -  R32).half())-R16
</code></pre>
</blockquote>
</aside>
<p>So, the added snippet is that float16 cannot handle the full range of float32, so I compare R16 and R32to16 and check whether R16 is the most accurate float16 expressing R32 by applying eps.</p>