What is the mathematical terms or idea for torch.isclose?

hungpham3112: <p>I’m finding a theoretical idea behind the formula in <code>torch.isclose()</code></p>
<p><code>∣input−other∣≤atol+rtol×∣other∣</code></p>
<p>Why they combine absolute tolerance and relative tolerance  together in this formula? I understand <code>|input -other|</code> in here is absolute error, but why <code>absolute error &lt;= atol + rtol x |other|</code>?</p>

tom: <p>I think it is mostly a convenience thing originating from the old <a href="https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style" rel="noopener nofollow ugc">10.0 times 0.1 is hardly ever 1.0</a>.<br>
I’d think about it in terms of “intuitively, close means absolute error &lt;= atol, but for large numbers we need something extra”.<br>
You can see that “are these equal up to numerical error”  typically needs a relative tolerance as well with an experiment setting rtol to 0 like this (might not work every time you draw random numbers, but very often, I tried a thousand times and it was invariably like this):</p>
<pre data-code-wrap="python"><code class="lang-python">In [1]: import torch
In [2]: a = torch.randn(1000, dtype=torch.float64)
In [3]: b = a.float().double()
In [4]: torch.isclose(a, b).all()
Out[4]: tensor(True)
In [5]: torch.isclose(a, b, rtol=0).all()
Out[5]: tensor(False)
</code></pre>

hungpham3112: <p>I prefer error analysis reason behind this formula to intuition. But anyway, thank you for your help.</p>

tom: <p>What I was trying to show, I guess, was that floating point representation is inherently with relative accuracy. And then you want an absolute term for when you encounter actual zeros.</p>

hungpham3112: <p>Do you have any official document to study about relative tolerance and absolute tolerance in this formula? It’s really hard to find a document out there. No one explain why choosing <em>rtol=1e-05</em>, <em>atol=1e-08</em></p>

tom: <aside class="quote no-group" data-username="hungpham3112" data-post="5" data-topic="195009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hungpham3112/48/66716_2.png" class="avatar"> hungpham3112:</div>
<blockquote>
<p>Do you have any official document to study about relative tolerance and absolute tolerance in this formula?</p>
</blockquote>
</aside>
<p>No. In PyTorch, it comes from numpy.isclose, which takes them from numpy.allclose which has had that since the first git commit or so…</p>
<p>Best regards</p>
<p>Thomas</p>

nicholdav: <p><a class="mention" href="/u/hungpham3112">@hungpham3112</a> like <a class="mention" href="/u/tom">@tom</a> said you’ll find more info if you look for questions about numpy.allclose, e.g. <a href="https://stackoverflow.com/questions/65909842/what-is-rtol-for-in-numpys-allclose-function" class="inline-onebox" rel="noopener nofollow ugc">python - What is rtol for in numpy's allclose function? - Stack Overflow</a></p>
<blockquote>
<p>I am confused about the point of <code>rtol</code>. Why not just use a single tolerance value <code>tol</code>, and if <code>|a-b| &lt; tol</code>, then return <code>False</code>? Obviously, following the above equation, I could just do this manually by setting <code>rtol</code> to zero, thereby making everything symmetric. What is the point of the symmetry-breaking <code>rtol</code> facto</p>
</blockquote>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/65909842/what-is-rtol-for-in-numpys-allclose-function/65909907#65909907">
  <header class="source">

      <a href="https://stackoverflow.com/questions/65909842/what-is-rtol-for-in-numpys-allclose-function/65909907#65909907" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2988730/mad-physicist" target="_blank" rel="noopener nofollow ugc">
    <img alt="Mad Physicist" src="https://i.stack.imgur.com/ZOX7t.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/65909842/what-is-rtol-for-in-numpys-allclose-function/65909907#65909907" target="_blank" rel="noopener nofollow ugc">What is rtol for in numpy's allclose function?</a>
</h4>

<div class="tags">
  <strong>python, numpy</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/2988730/mad-physicist" target="_blank" rel="noopener nofollow ugc">
    Mad Physicist
  </a>
  on <a href="https://stackoverflow.com/questions/65909842/what-is-rtol-for-in-numpys-allclose-function/65909907#65909907" target="_blank" rel="noopener nofollow ugc">10:02PM - 26 Jan 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>You generally want to use <code>rtol</code>: since the precision of numbers and calculations is very much finite, larger numbers will almost always be less precise than smaller ones, and the difference scales linearly (again, in general). The only time you use <code>atol</code> is for numbers that are so close to zero that rounding errors are liable to be larger than the number itself.</p>
<p>Another way to look at it is <code>atol</code> compares fixed decimal places, while <code>rtol</code> compares significant figures.</p>
</blockquote>
<p>The defaults were probably chosen because those are typical values people care about in scientific/engineering applications</p>