Torch.finfo() eps weird behavior

Harimus: <p>Hello, so I think showing the problematic behavior I’ve encountered first is best:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a135ce8af27e35e89d2f014f85dbf1b14d9b1c8.png" data-download-href="https://discuss.pytorch.org/uploads/default/4a135ce8af27e35e89d2f014f85dbf1b14d9b1c8" title="Screenshot from 2021-05-19 19-01-11"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a135ce8af27e35e89d2f014f85dbf1b14d9b1c8_2_690x253.png" alt="Screenshot from 2021-05-19 19-01-11" data-base62-sha1="aziOHcmnMNRYqeNtKZi6odA1HDi" width="690" height="253" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a135ce8af27e35e89d2f014f85dbf1b14d9b1c8_2_690x253.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a135ce8af27e35e89d2f014f85dbf1b14d9b1c8_2_1035x379.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a135ce8af27e35e89d2f014f85dbf1b14d9b1c8.png 2x" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-05-19 19-01-11</span><span class="informations">1051×386 24 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So I’ve been training Policy-gradient (Reinforcement Learning) methods with <em>TanhDistributions</em>. Tanh distributions built-in <em>log_prob()</em> method is very sensitive to the numeric value close to 1 &amp; -1. (giving NaN for 1.0 &amp; -1.0, which often crashes the entire program) for float32 this means that sometime I need to <code>torch.clamp</code> the input, with an epsilon. I saw this weird thing during debugging that.</p>
<p>According to the <a href="https://pytorch.org/docs/stable/type_info.html" rel="noopener nofollow ugc">Type info documentation</a> <em>eps</em> is supposed to be the smallest number where <code>1.0 + eps != 1.0</code>, which then makes me believe that <code>x * eps</code> with<code> x &lt; 1</code> should result in a <code>False</code> for this equation. Yet this happen only first at 0.5, and <em>somehow</em> this only happens at <code>1.0 + 0.5*eps !=1.0</code> and somehow, not for <code>1.0 - 0.5*eps != 1.0</code>.</p>
<p>Am I just misunderstanding what <em>eps</em> is supposed to be here?</p>

KFrank: <p>Hi Dan!</p>
<aside class="quote no-group" data-username="Harimus" data-post="1" data-topic="121732" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harimus/48/38475_2.png" class="avatar"> Harimus:</div>
<blockquote>
<p><em>eps</em> is supposed to be the smallest number where <code>1.0 + eps != 1.0</code></p>
</blockquote>
</aside>
<p>This is a documentation bug.  Quoting from pytorch’s <a href="https://pytorch.org/docs/1.8.1/type_info.html" rel="noopener nofollow ugc">1.8.1 documentation</a>:</p>
<pre><code class="lang-plaintext">eps     float     The smallest representable number
                  such that 1.0 + eps != 1.0.
</code></pre>
<p>This is sort of on the right track, but really isn’t correct.</p>
<p><a href="https://numpy.org/doc/1.20/reference/generated/numpy.finfo.html" rel="noopener nofollow ugc">Numpy’s (1.20) documentation</a> gets it right:</p>
<pre><code class="lang-plaintext">eps : float

    The difference between 1.0 and the next smallest
    representable float larger than 1.0. For example,
    for 64-bit binary floats in the IEEE-754 standard,
    eps = 2**-52, approximately 2.22e-16.
</code></pre>
<p>What’s at issue in your numerical test is that neither <code>1.0 + 0.5 * eps</code><br>
nor <code>1.0 + 0.9 * eps</code> is exactly representable, so the floating-point<br>
arithmetic has to decide whether to round up or down to a representable<br>
number.</p>
<p>In the first case, it rounds down to <code>1.0</code>; in the second, up to <code>1.0 + eps</code>.<br>
(By definition, there are no representable numbers in between.)</p>
<p>Here’s an illustration that shows what is going on including the correctness<br>
of numpy’s definition of <code>eps</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.tensor ([1.0]) + 0.5000001 * torch.finfo().eps == 1.0
tensor([False])
&gt;&gt;&gt; torch.tensor ([1.0]) + 0.5000000 * torch.finfo().eps == 1.0
tensor([True])
&gt;&gt;&gt; (torch.tensor ([1.0]) + 0.5000001 * torch.finfo().eps) - 1.0 == torch.finfo().eps
tensor([True])
&gt;&gt;&gt; (torch.tensor ([1.0]) + 0.5000000 * torch.finfo().eps) - 1.0 == torch.finfo().eps
tensor([False])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Harimus: <p>Aaah now I get it. The numpy description was a bit more clear to me.</p>
<p>Thank you for the thorough answer KFrank!</p>