Float Overflow?

Arie_Cattan: <p>Pytorch version 1.3.0</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc1dc284da855ab1f1433fe8bab51fd1fe1cb4b5.png" alt="Screen Shot 2020-02-10 at 14.47.39" data-base62-sha1="t7HaaVKdkhLkw0icd6sRr9kD421" width="513" height="75"></p>

Tony-Y: <pre><code class="lang-auto">&gt;&gt;&gt; torch.tensor(100000016.0, dtype=float) == torch.tensor(100000015.0, dtype=float)
tensor(False)
</code></pre>

ptrblck: <p>Some additional context:<br>
You are creating these “integer” tensors using the default type of <code>torch.float32</code>.<br>
As seen in the <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">Wikipedia article to IEEE FP32</a>, integer numbers <code>&gt;2**26</code> should be rounded to a multiple of 8.</p>
<p><a class="mention" href="/u/tony-y">@Tony-Y</a> is in fact creating these numbers as <code>torch.float64</code>, which can be checked via:</p>
<pre><code class="lang-python">a = torch.tensor(100000016.0, dtype=float)
print(a.type())
&gt; torch.DoubleTensor
</code></pre>
<p>which has a higher limit for these rounding errors.</p>

wqm: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82251ed67a1b9fd885411b31b9ac508ee8535085.png" data-download-href="https://discuss.pytorch.org/uploads/default/82251ed67a1b9fd885411b31b9ac508ee8535085" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82251ed67a1b9fd885411b31b9ac508ee8535085_2_392x500.png" alt="image" data-base62-sha1="izjBhH5GdKIYUNwgZ7MHJA5bILX" width="392" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82251ed67a1b9fd885411b31b9ac508ee8535085_2_392x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/2/82251ed67a1b9fd885411b31b9ac508ee8535085.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/2/82251ed67a1b9fd885411b31b9ac508ee8535085.png 2x" data-dominant-color="272827"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">496×632 38.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>As described in the linked Wikipedia page:<br>
"Integers between <code>2**n</code> and <code>2**(n+1)</code> round to a multiple of <code>2**(n-23)</code>"</p>
<p>Since your value is between <code>2**24</code> and <code>2**25</code>, the result will thus be rounded to multiples of <code>2</code>.</p>
<pre><code class="lang-python">a = 16799999
print(2**24 &lt; a)
&gt; True
print(a &lt; 2**25)
&gt; True

print(torch.tensor(a).float()) # 16799999
&gt; tensor(16800000.) # rounded up

print(torch.tensor(a-1).float()) # 16799998
&gt; tensor(16799998.) # representable

print(torch.tensor(a+2).float()) # 16800001 
&gt; tensor(16800000.) # rounded down

print(torch.tensor(a+3).float()) # 16800002
&gt; tensor(16800002.) # representable
</code></pre>

wqm: <p>Thank you for your reply. With the help of your explanation and wiki link, I have understood now ~</p>