How can I show that one function is more numerically stable than another equivalent function?

username: knownad
cooked: <p>Hi, I’m looking for a suggestion. Suppose I have two equivalent functions, say softmax versus the softmax with max subtracting trick (which is more numerical stable). How would you demonstrate (e.g., in a paper) that one version is more numerically stable than the other?</p>

username: albanD
cooked: <p>I guess you would need to check that usual range of each variable. And for each operation you perform measure the difference between the largest and smallest value. This will give you the precision of the op. Then you can try and compute the worst case scenario for your function.</p>
<p>I’m sure you can find some material on google from people studying these subjects for differential equation solvers.</p>

username: KFrank
cooked: <p>Hi Known!</p>
<aside class="quote no-group" data-username="knownad" data-post="1" data-topic="58812" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a698b9/48.png" class="avatar"> knownad:</div>
<blockquote>
<p>Hi, I’m looking for a suggestion. Suppose I have two equivalent functions, say softmax versus the softmax with max subtracting trick (which is more numerical stable). How would you demonstrate (e.g., in a paper) that one version is more numerically stable than the other?</p>
</blockquote>
</aside>
<p>In the context of functions, I would prefer to use the terms “more<br>
accurate” and “less accurate.”*</p>
<p>The simplest way to compare the accuracy of various versions of<br>
your function is to compare them to the exact (or at least a known,<br>
much more accurate) result.</p>
<p>As a practical matter (but maybe not fully satisfying to a purist),<br>
you can compare your results with those calculated using greater<br>
precision.  So, if your two implementations use 32-bit floating-point<br>
arithmetic (<code>dtype = float32</code>), then you can also compute your<br>
results using 64-bit arithmetic (<code>dtype = float64</code>) and treat those<br>
as the “exact” results.  Your (32-bit) version that comes closest<br>
to the 64-bit result is the more accurate.</p>
<p>If you have access to an extended-precision / arbitrary-precision<br>
math library, you can take this approach further (or apply it to<br>
your 64-bit versions, etc.).</p>
<p>I would be comfortable using a 64-bit comparison to demonstrate the<br>
greater accuracy of an improved 32-bit algorithm (but maybe that’s<br>
just good-enough-for-ranch-work me), and I would certainly accept a<br>
comparison with the results of a series of increasingly more precise<br>
extended-precision computations.</p>
<p>*)  “Numerical stability” has a pretty well understood connotation in<br>
a variety of contexts.  But when being used precisely, it is most<br>
commonly used in the context of iterative algorithms:  Unstable<br>
algorithms have iterations that amplifies errors, while stable algorithms<br>
have iterations that reduce errors.  So, unless you are using an<br>
iterative algorithm to compute your function, I would prefer to speak<br>
of accuracy rather than stability.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: knownad
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="58812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>I’m sure you can find some material on google from people studying these subjects for differential equation solvers.</p>
</blockquote>
</aside>
<p>Thank you for the pointers!</p>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="58812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I would prefer to use the terms “more<br>
accurate” and “less accurate.”*</p>
</blockquote>
</aside>
<p>Thank you! I think I’m actually looking at both, accuracy and also as an iterative algorithm that amplifies errors, as you mentioned here:</p>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="58812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Unstable algorithms have iterations that amplifies errors, while stable algorithms have iterations that reduce errors.</p>
</blockquote>
</aside>
<p>Is there a reference about performing many consecutive matrix multiplications amplifying errors?</p>
<p>BTW, I’m checking out Trefethen’s book “Numerical Linear Algebra”, Lecture 14 “Stability”.</p>