Computing the Hessian matrix of a function w.r.t a tensor

username: AndreaRossi
cooked: <p>Hi everyone, I really need some help with the following problem.</p>
<p>I have a list of embeddings [e_1, e_2… e_k].<br>
Each item e_i in the list is a monodimensional Tensor with size, say, 200.</p>
<p>I have successfully trained my embeddings in the following setting:</p>
<ul>
<li>I have a scoring function that takes as an input 3 embeddings: φ(e_i, e_j, e_k)</li>
<li>I have a training set that is a set of triples (item_i, item_j, item_k), where item_i, item_j and item_k correspond to the embeddings e_i, e_j, e_k</li>
<li>In training, I initialize my embeddings randomly and then train them to minimize the values of φ for the triples in my training set.</li>
</ul>
<p>I have completed the training and it worked fine, so that part is ok (yay!).</p>
<p>What I am now trying to do is to compute the Hessian matrix for a specific embedding in the list.<br>
To be more specific, given a specific embedding e_x:</p>
<ul>
<li>I can isolate all triples featuring item_x from the training set, let’s call them x_triples</li>
<li>I define f = sum([phi(a, b, c) for (a, b, c) in x_triples])</li>
<li>I would like to compute the Hessian matrix of f with respect to e_x</li>
</ul>
<p>I said that e_x is a monodimensional vector of size 200, so it is like<br>
e_x = [x1, x2, x3 … x200]</p>
<p>The Hessian matrix should be a 200x200 matrix that, in row i and column j, contains the second order partial derivative of f with respect to x_i and x_j.<br>
In other words, something like this (the image is shamelessly taken from wikipedia but it should make things a bit clearer hopefully).</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dea0aca6815c2515c7355b56190c08bab78d869.png" alt="image" data-base62-sha1="6yaUZh151uEP8Af65YibLWMyKA9" width="270" height="189"></p>
<p>This leads me to the actual question.</p>
<p>I see in this page that there is a “torch.autograd.functional.hessian” function that may be the way to go for my scenario.<br>
<a href="https://pytorch.org/docs/stable/autograd.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html</a></p>
<p>I see from the documentation that its main parameters are:</p>
<ul>
<li><strong>func</strong> (<em>function</em>) – a Python function that takes Tensor inputs and returns a Tensor with a single element.</li>
<li><strong>inputs</strong> (<em>tuple of Tensors</em> <em>or</em> <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a>) – inputs to the function <code>func</code>.</li>
</ul>
<p>I would say that in my case :</p>
<ul>
<li>func is be the sum of all phi(a, b, c) for any (a, b, c) in x_triples</li>
<li>inputs is e_x</li>
</ul>
<p>My problem is that all embeddings e1, e2… ek are used by func, but I only want the Hessian to be computed respect to e_x (that is, its 200 components).<br>
Is there a way to do that?</p>
<p>(Sorry for the long post)</p>

username: AlphaBetaGamma96
cooked: <p>Do you some code to reproduce the issue?</p>
<ul>
<li>Also, how exactly does <code>e_x</code> relate to <code>x_triples</code>?</li>
<li>I assume you want to compute the hessian for each embedding vector <code>e_i</code>? So, you’d have an output of <code>k</code> hessian matrices?</li>
<li>Also, are the embeddings <code>e_i</code>, <code>e_j</code>, and <code>e_k</code> three random embeddings from the list of embeddings stated at the beginning or three components of the an embedding <code>e_i</code>?</li>
</ul>

username: AndreaRossi
cooked: <p>Hi, <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>, thanks for replying!<br>
I don’t have any code to reproduce the issue, because I’m still trying to figure out how to write the hessian part <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>I think my previous post was a bit confusing, sorry about that. I’ll try to make it more understandable.</p>
<p>Let’s say I’m computing embeddings of words, and my input is made entirely of trigrams (triples of words).</p>
<p>I have a function φ that can compute the likelihood of a trigram using the embeddings of the corresponding words.</p>
<p>What I am trying to do is to compute the Hessian matrix of the likelihood of the trigrams that contain a certain word, with respect to the embedding of that word.</p>
<p>So <strong>item_x</strong> is the word I want to get the Hessian with respect to; <strong>e_x</strong> is its trained embedding; <strong>x_triples</strong> is the list of trigrams that feature the word <strong>item_x</strong>.<br>
I hope this clarifies your first question, so I move to the others.</p>
<p>Second question: no, actually I just need it to run on a specific word at a time, so for one embedding.<br>
It is for an explainability project, so I only need it for the specific embedding I am trying to explain (I won’t try to explain them all).</p>
<p>Third question: in mu example e_i, e_j, and e_k were meant to be three separate embeddings.<br>
Each of them was supposed to be monodimensional and with size 200.</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="AndreaRossi" data-post="3" data-topic="116621">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrearossi/48/28336_2.png" class="avatar"> AndreaRossi:</div>
<blockquote>
<p>What I am trying to do is to compute the Hessian matrix of the likelihood of the trigrams that contain a certain word, with respect to the embedding of that word.</p>
</blockquote>
</aside>
<p>NLP tasks are a little out of my skill set, but what I’d suggest you try is figure out how exactly the embedding of given word is link to the likelihood of the trigrams, and build a function which takes the embeddings and returns the likelihood. Then you can just use the <code>torch.autograd.functional.hessian</code> to calculate it!</p>