GELU Pytorch formula?

username: Sam_Lerman
cooked: <p>Hi, for a particular reason, I need to hand-derive a derivative for a GELU and it has to match the GELU implementation in Pytorch, but I tried finding the exact GELU formula and I’m a little confused. Could someone help simplify it for me by confirming whether the following is exactly equivalent to <code>torch.nn.GELU</code>:</p>
<p><code>0.5 * x * (1 + torch.tanh(np.math.sqrt(2 / np.math.pi) * (x + 0.044715 * torch.pow(x, 3)))</code></p>
<p>If not, could you provide the correct explicit formula? (Not for the derivative, just for the raw GELU)</p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/pytorch/pytorch/blob/b7f4b6a6de30116f1b44a08fab9499dd5eb2de7d/test/cpp/api/functional.cpp#L839-L845">this test</a>, <code>gelu</code> should correspond to:</p>
<pre><code class="lang-python">const auto y_exp = x * 0.5 * (1.0 + torch::erf(x / std::sqrt(2.0)));
</code></pre>
<p><code>torch::erf</code> is probably used from the <code>std</code> lib or the CUDA implementation.</p>

username: Sam_Lerman
cooked: <p>I saw that elsewhere, but it doesn’t help me compute a derivative by hand because I don’t know the exact function that torch::erf takes.</p>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="Sam_Lerman" data-post="3" data-topic="79875" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/48/37165_2.png" class="avatar"> Sam_Lerman:</div>
<blockquote>
<p>I saw that elsewhere, but it doesn’t help me compute a derivative by hand because I don’t know the exact function that torch::erf takes.</p>
</blockquote>
</aside>
<p>The function can be found in the documentation for <a href="https://pytorch.org/docs/stable/torch.html#torch.erf" rel="noopener nofollow ugc">torch.erf()</a>.</p>
<p>More detail can be found in the wikipedia entry <a href="https://en.wikipedia.org/wiki/Error_function" rel="noopener nofollow ugc">Error function</a>.</p>
<p>As you can see, its derivative is just the probability density function<br>
for the normal distribution.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Sam_Lerman
cooked: <p>Hmm, but how does Pytorch implement that integral? Is there no simple function equivalent to Pytorch’s implementation you could provide that I could enter into the following website:</p>
<p><a href="http://www.matrixcalculus.org" class="onebox" target="_blank" rel="nofollow noopener">http://www.matrixcalculus.org</a></p>
<p>I need to derive an MLP that uses GeLU activations and it would be convenient if I could just enter the full formula into that tool above.</p>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="Sam_Lerman" data-post="5" data-topic="79875" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/48/37165_2.png" class="avatar"> Sam_Lerman:</div>
<blockquote>
<p>Is there no simple function equivalent to Pytorch’s implementation you could provide</p>
</blockquote>
</aside>
<p><code>erf()</code> is a so-called <em>special function.</em>  It is well studied, well<br>
understood, and (reasonably) easy to calculate with modern<br>
numerical analysis techniques.</p>
<p>But there is no “simple function equivalent” to it (other than <code>erf()</code><br>
or things like the integral used to define it).  So <code>erf()</code> is the best<br>
you’ve got.</p>
<aside class="quote no-group">
<blockquote>
<p><a href="http://www.matrixcalculus.org" rel="noopener nofollow ugc">http://www.matrixcalculus.org</a></p>
<p>it would be convenient if I could just enter the full formula into that tool above.</p>
</blockquote>
</aside>
<p>If you want to “compute a derivative by hand,” that’s easy, because<br>
the derivative of <code>erf()</code> is an <em>elementary function,</em> namely the<br>
normal distribution.</p>
<p>My pocket calculator doesn’t know how anything about <code>erf()</code>, so<br>
there’s no practical way for me to calculate <code>erf()</code> with my calculator.<br>
But python does know about <code>erf()</code>, so <code>math.erf (1.234)</code> works<br>
just fine in python.</p>
<p>If the calculus tool you linked to knows about <code>erf()</code>, then you should<br>
be good.  If it doesn’t, you’ll have to “compute a derivative by hand”<br>
(or switch to a better calculus tool).</p>
<p>Best.</p>
<p>K. Frank</p>

username: Sam_Lerman
cooked: <p>I’m confused, if there is no function equivalent, then how does Pytorch compute it? From reading about GeLU, it seems like they used an approximation of the erf function in the paper using tanh.</p>