Arbitrary-floating numbers and automatic differenation

zacmon: <p>Hi,</p>
<p>I have a likelihood function in which if I have a data point which is, say, 68, I must then calculate 68 derivatives. I have hard-coded a decent amount of derivatives already. Because of the nature of the likelihood function and how I must use a ridiculous order of derivatives, I need an arbitrary floating point library for which I use mpmath. I’ve tried optimizing my code by dynamically switching the precision, using parallel python or gnu parallel and HPCs, etc. I’m debating on whether or not I should migrate to C++ to get the calculation even quicker. Anyway, my question, does pytorch enable the use of arbitrary floating point numbers in its automatic differentiation? I’d like to experiment with the idea of staying in python, automatic differentiation, and employing GPUs to speed up my own workflow and optimize computations. Also, does pytorch’s JIT allow for mpmath or another arbitrary floating precision module? For reference, I don’t think Numba does allow arb floating precision numbers.</p>
<p>Thanks,<br>
Zach</p>

KFrank: <p>Hi Zach!</p>
<aside class="quote no-group" data-username="zacmon" data-post="1" data-topic="67674" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/db5fbb/48.png" class="avatar"> zacmon:</div>
<blockquote>
<p>does pytorch enable the use of arbitrary floating point numbers in its automatic differentiation?</p>
</blockquote>
</aside>
<p>No, pytorch’s autograd framework uses pre-evaluated analytic*<br>
differentiation, not numerical differentiation.</p>
<p>If you can figure out how to write your likelihood function using<br>
pytorch tensor operations, then autograd will differentiate your<br>
function without additional work on your part.</p>
<p>autograd uses subclasses of <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" rel="noopener nofollow ugc">torch.autograd.Function</a>.  These<br>
have a <code>forward()</code> method that calculates the function itself,<br>
and a <code>backward()</code> method that calculates the gradient.  So,<br>
when the pytorch implementers wrote <code>torch.sin()</code>, they called<br>
some math library to compute <code>sin()</code>, they analytically differentiated<br>
<code>sin()</code> to get <code>cos()</code>, and then they called some math library to compute<br>
<code>cos()</code>.  (Of course, the math library computes <code>cos()</code> numerically,<br>
but the fact that <code>cos()</code> is the derivative of <code>sin()</code> was determined<br>
analytically.)</p>
<p>If you can’t figure out how to write your likelihood function using<br>
pytorch tensor functions, you will have to write your own <code>backward()</code><br>
function for it.  (This is perfectly doable, but you’re better off using<br>
the existing pytorch tensor functions, if you can.)</p>
<p>*)  Nothing prevents an implementation from using numerical<br>
differentiation in a <code>backward()</code> function – all <code>backward()</code> needs<br>
to do is return the correct gradient.  The autograd framework neither<br>
knows nor cares how the gradient was computed.  I’m just not aware<br>
of any built-in pytorch function that uses numerical differentiation for<br>
its <code>backward()</code>.  So you could write a <code>backward()</code> function that uses<br>
numpy or mpmath or whatever to perform its numerical differentiation,<br>
provided that you repackage the final result you return as a pytorch<br>
tensor.</p>
<p>(Note, pytorch offers <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.gradcheck" rel="noopener nofollow ugc">torch.autograd.gradcheck()</a> for checking<br>
gradients numerically.)</p>
<p>Best.</p>
<p>K. Frank</p>

zacmon: <p>Thanks for the reply! I’m just getting introduced to autodiff in general. Do you think autodiff would outperform hard-coding a function for, say, a 60th derivative which uses high-order chain rule? Because of Faa di Bruno’s formula, lower-order derivatives are used over and over again in high-order chain rule calculations. But I’m guessing pytorch has an optimized higher-order chain rule implementation? E.g. in Faa di Bruno’s formula, it seems the bottleneck is unavoidable in that so many addition and multiplication operations are applied for higher and higher-order chain rule derivatives. I suppose the use of GPU via pytorch would speed this up. But is there someone familiar with higher-order chain rule and pytorch?</p>

KFrank: <p>Hello Zach!</p>
<aside class="quote no-group quote-modified" data-username="zacmon" data-post="3" data-topic="67674" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/db5fbb/48.png" class="avatar"> zacmon:</div>
<blockquote>
<p>Do you think autodiff would outperform hard-coding a function for, say, a 60th derivative which uses high-order chain rule?</p>
<p>…</p>
<p>But is there someone familiar with higher-order chain rule and pytorch?</p>
</blockquote>
</aside>
<p>I don’t really understand what your use case is.</p>
<p>However, you should understand that pytorch is a neural-network<br>
framework, and not a general-purpose automatic-differentiation<br>
package.</p>
<p>Pytorch’s purpose in life is to support building and training neural<br>
networks.  A neural network is trained by (partially) minimizing a<br>
loss function with respect to the network’s parameters.  For a<br>
variety of reasons, this minimization is performed using various<br>
gradient-descent algorithms.  To this end pytorch’s autograd<br>
framework calculates the gradient (first-derivative) of the loss<br>
with respect to the network’s parameters.</p>
<p>autograd is a very useful specialized tool for pytorch’s use case,<br>
but it is not particularly general-purpose.</p>
<p>There is no direct support in autograd for higher-order derivative.<br>
There is no support in autograd for finding the derivative of a function<br>
for whose pieces you haven’t already provided derivatives (as<br>
<code>backward()</code> methods).</p>
<p>Note, to clarify the comment in my previous post about “analytic”<br>
differentiation:</p>
<p>Think of a neural network as a bunch of “simple” functions that<br>
are composed with one another – that is, chained together.<br>
autograd differentiates (finds the gradient of) such a function.<br>
To do so, it relies on the authors of the simple functions to<br>
provide their derivatives, packaged in the <code>backward()</code> method.<br>
One imagines that those authors calculated those derivatives<br>
analytically, but they didn’t have to.</p>
<p>autograd then implements the chain rule <em>numerically</em> to differentiate<br>
the big, composite function, using the derivative of its pieces obtained<br>
from their <code>backward()</code> methods.</p>
<p>If you think this through, you will see that autograd is great for pytorch’s<br>
use case, but probably not very useful for what I think you might want<br>
to do.</p>
<p>Good luck!</p>
<p>K. Frank</p>