3rd order partial derivatives of scalar function w.r.t. input matrix

rubencart: <p>I want to compute 3rd order partial derivatives of a scalar function w.r.t. an input matrix. So if the input matrix X is 5x5 (so 25 entries), I think I should end up with (5x5)x(5x5)x(5x5) entries (25^3), right?</p>
<p><strong>Main question</strong>: Is this a correct and if so, the right/best/fastest/… way to do it?</p>
<pre><code class="lang-auto">def jacobian(func, inp):
    return torch.autograd.functional.jacobian(func, inp, create_graph=True)

def third_partial_derivatives(scalar_func, inp):
    return jacobian(
        lambda x: jacobian(
            lambda y: jacobian(scalar_func, y), 
            x
        ), 
        inp
    )
</code></pre>
<p><strong>Side question</strong>: I noticed that the 2nd order derivatives already contain almost exclusively very small values, and the 3rd order derivatives contain almost only zeroes. Could this be caused by underflow, and is there a way I could fix this?</p>
<p>Thanks in advance!</p>

AlphaBetaGamma96: <p>Would you mind sharing what scalar function you’re using for your input matrix? Is it the determinant?</p>

rubencart: <p>It’s the log partition function of a CRF, computed with a differentiable dynamic programming algorithm with <a href="https://github.com/harvardnlp/pytorch-struct" rel="noopener nofollow ugc">torch-struct</a>, which I want to differentiate (3x) w.r.t. input log edge/node potentials <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I can give you more details but I don’t know if they’ll be relevant?</p>

AlphaBetaGamma96: <p>Having a bit of information on the function might help a little bit with giving me an idea as to what you’re differentiating.</p>
<p>I have done calculating the 2nd derivative of a determinant function (which is why I asked if it’s that). When I made a custom version of that function it also suffered from near 0 values. Which I solved by using a custom <code>torch.autograd.Function</code> and manually defining the derivatives (with some <code>torch.linalg.svd</code> as well in order to handle singular matrices).</p>
<p>Could this be a potential avenue for you?</p>

rubencart: <p>Thanks for the suggestion :-). I’d prefer not to define the derivatives myself though, I think they could be quite complex, which is exactly why I’d like to make use of the automatic differentiation of pytorch.</p>
<p>The first and main question is whether repeating <code>torch.autograd.functional.jacobian</code> like in my post theoretically gives me the correct 3rd order derivatives?</p>
<p>The 2nd question, the underflow problem, I could perhaps solve by giving <code>lambda x: torch.exp(scalar_func(x))</code> as argument instead of <code>scalar_func</code>, since <code>scalar_func</code> returns the log of the partition function? Or does this not make sense, for numerical stability?</p>

rubencart: <p><a class="mention" href="/u/tom">@tom</a> or <a class="mention" href="/u/smth">@smth</a> or <a class="mention" href="/u/ptrblck">@ptrblck</a> or <a class="mention" href="/u/alband">@albanD</a> could someone help me? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="rubencart" data-post="5" data-topic="133165">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rubencart/48/25961_2.png" class="avatar"> rubencart:</div>
<blockquote>
<p>The first and main question is whether repeating <code>torch.autograd.functional.jacobian</code> like in my post theoretically gives me the correct 3rd order derivatives?</p>
</blockquote>
</aside>
<p>I think this should give the correct results, althought you could check this with doing the Jacobian call twice and checking with <code>torch.autograd.functional.Hessian</code> (although Hessian doesn’t have batch-support, so you’d have to apply it iteratively)</p>
<aside class="quote no-group" data-username="rubencart" data-post="5" data-topic="133165">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rubencart/48/25961_2.png" class="avatar"> rubencart:</div>
<blockquote>
<p>The 2nd question, the underflow problem, I could perhaps solve by giving <code>lambda x: torch.exp(scalar_func(x))</code> as argument instead of <code>scalar_func</code>, since <code>scalar_func</code> returns the log of the partition function? Or does this not make sense, for numerical stability?</p>
</blockquote>
</aside>
<p>Applying the <code>exp</code> of a <code>log</code> value can aid the stability but it generally does become function specific. For example, in my work I have the ratio of two quantities and I can decompose that into the <code>exp</code> of a substraction within the log-domain.</p>