First logarithm and then exponential, what's the reason?

username: Ali_Akyurek
cooked: <p>Hi all,<br>
I have a question regarding PositionalEncoder implementaton in official transformers tutorial <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">*</a><br>
There’s the following implementation there:</p>
<pre><code class="lang-auto">torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
</code></pre>
<p>However I’d expect to see something like</p>
<pre><code class="lang-auto">1/((10000.0 ** (torch.arange(0, d_model, 2) / d_model)))
</code></pre>
<p>What’s the advantage of first "log"ing and then "exp"onenting?</p>

username: KFrank
cooked: <p>Hi Ali!</p>
<aside class="quote no-group" data-username="Ali_Akyurek" data-post="1" data-topic="153137" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_akyurek/48/50173_2.png" class="avatar"> Ali_Akyurek:</div>
<blockquote>
<p>There’s the following implementation there:</p>
<pre><code class="lang-auto">torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
</code></pre>
<p>However I’d expect to see something like</p>
<pre><code class="lang-auto">1/((10000.0 ** (torch.arange(0, d_model, 2) / d_model)))
</code></pre>
<p>What’s the advantage of first "log"ing and then "exp"onenting?</p>
</blockquote>
</aside>
<p>There is a performance advantage (but no substantive numerical<br>
advantage). The <code>pow()</code> (<code>**</code>) function is modestly more expensive<br>
than <code>exp()</code>.  (While <code>log()</code> is expensive, you only have to compute<br>
it for a single scalar, and not the whole <code>arange()</code> tensor.)</p>
<p>For timing purposes I have made a modest performance improvement<br>
to your <code>**</code> version by getting rid of the reciprocal (<code>1 / x</code>) computation.<br>
On my system – for large tensors – the “exp-log” version displays a<br>
systematic performance advantage both on the cpu and gpu.</p>
<p>Here is a timing script:</p>
<pre data-code-wrap="python"><code class="lang-python">import time
import math

import torch
print (torch.__version__)
print (torch.version.cuda)
print (torch.cuda.get_device_name())

n = 10

for  device in ('cpu', 'cuda'):
    #    
    # exp-log version
    #
    for  d_model in (10000000, 100000000):
        rng = torch.arange (0, d_model, 2).to (device)
        # warm up
        torch.cuda.synchronize()
        for  i in range (3):
            xExp = torch.exp (rng * (-math.log (10000.0) / d_model))
        # time
        torch.cuda.synchronize()
        start = time.time()
        for  i in range (n):
            xExp = torch.exp (rng * (-math.log (10000.0) / d_model))
        torch.cuda.synchronize()
        t = (time.time() - start) / n
        print ('exp-log, %s:  d_model = %d, t = %f' % (device, d_model, t))
    #
    # pow version (avoid reciprocal)
    #
    for  d_model in (10000000, 100000000):
        rng = torch.arange (0, d_model, 2).to (device)
        # warm up
        torch.cuda.synchronize()
        for  i in range (3):
            xPow = 10000.0 ** (-rng / d_model)
        # time
        torch.cuda.synchronize()
        start = time.time()
        for  i in range (n):
            xPow = 10000.0 ** (-rng / d_model)
        torch.cuda.synchronize()
        t = (time.time() - start) / n
        print ('pow, %s:  d_model = %d, t = %f' % (device, d_model, t))
    #
    print ('torch.allclose (xExp, xPow) =', torch.allclose (xExp, xPow))
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.10.0
10.2
GeForce GTX 1050 Ti
exp-log, cpu:  d_model = 10000000, t = 0.011005
exp-log, cpu:  d_model = 100000000, t = 0.108074
pow, cpu:  d_model = 10000000, t = 0.035099
pow, cpu:  d_model = 100000000, t = 0.365038
torch.allclose (xExp, xPow) = True
exp-log, cuda:  d_model = 10000000, t = 0.001014
exp-log, cuda:  d_model = 100000000, t = 0.010088
pow, cuda:  d_model = 10000000, t = 0.002260
pow, cuda:  d_model = 100000000, t = 0.020973
torch.allclose (xExp, xPow) = True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Ali_Akyurek
cooked: <p>Wow! Thank you very much K. Frank for taking time to tell the background and to implement the proof.<br>
My assumption was that it might be due to a numerical stability improvement. But as your answer says, this is performance related.<br>
Attaching my results:</p>
<pre><code class="lang-auto">1.11.0+cu113
11.3
Quadro P2000
exp-log, cpu:  d_model = 10000000, t = 0.015999
exp-log, cpu:  d_model = 100000000, t = 0.107793
pow, cpu:  d_model = 10000000, t = 0.029426
pow, cpu:  d_model = 100000000, t = 0.278817
torch.allclose (xExp, xPow) = True
exp-log, cuda:  d_model = 10000000, t = 0.001003
exp-log, cuda:  d_model = 100000000, t = 0.012005
pow, cuda:  d_model = 10000000, t = 0.003000
pow, cuda:  d_model = 100000000, t = 0.022995
torch.allclose (xExp, xPow) = True
</code></pre>