Jacobian with respect to a symmetric tensor

BruceWayne: <p>Hello,</p>
<p>I wanted to perform gradient operation of a 3-by-3 tensor with respect to another 3-by-3 tensor, which outputs a 3-by-3-by-3-by-3 tensor, see the following example code:</p>
<pre><code class="lang-auto">X = torch.tensor([[1,3,5],[3,1,7],[5,7,1]],dtype=torch.double)
X.requires_grad_(True)

def computeY(input):return torch.pow(input, 2)

dYdX = torch.autograd.functional.jacobian(computeY, X) 
</code></pre>
<p>This does exactly what the Jacobian operation does, however, it does not seem to take into consideration that <code>X</code> is symmetric.<br>
If <code>X</code> is a general matrix without symmetry information, then for example, dY[1,2]/dX[2,1] = 0. This is exactly what autograd does by default.<br>
If <code>X</code> is a symmetric matrix such that X[i,j] = X[j,i], then for example, dY[1,2]/dX[2,1] = dY[2,1]/dX[2,1] = dY[1,2]/dX[1,2] = 2<em>X[1,2] = 2</em>X[2,1] = 6.  Does anyone know if there’s anyway to provide such symmetry information to autograd?</p>
<p>Many thanks!</p>

KFrank: <p>Hi Bruce!</p>
<aside class="quote no-group" data-username="BruceWayne" data-post="1" data-topic="195596" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/d07c76/48.png" class="avatar"> BruceWayne:</div>
<blockquote>
<p>Does anyone know if there’s anyway to provide such symmetry information to autograd?</p>
</blockquote>
</aside>
<p>I don’t believe that there is any way to “tell” autograd that some matrix<br>
is to be understood to be a symmetric matrix.  You can, however, rewrite<br>
<code>computeY()</code> to treat its <code>input</code> as symmetric.</p>
<p>This may or may not fit your use case, but consider this illustrative script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

X = torch.tensor ([[1, 3, 5], [3, 1, 7], [5, 7, 1]], dtype = torch.double)
X_tril = X.tril()   # represent symmetric matrix by its lower triangle

print (X_tril)

def computeY (input) :   # assume input is lower-triangular
    input_sym = input.tril() + input.T.triu (1)   # build explicitly symmetric representation of input
    return torch.pow (input_sym, 2)

dYdX = torch.autograd.functional.jacobian (computeY, X_tril)   # only lower triangle of dX part is non-zero

print (dYdX[1, 2, 1, 2])   # X_tril[1, 2] is ignored (and is zero)
print (dYdX[1, 2, 2, 1])   # X_tril[2, 1] is used for both input_sym[2, 1] and input_sym[1, 2]
</code></pre>
<p>Here is its output:</p>
<pre><code class="lang-plaintext">2.1.2
tensor([[1., 0., 0.],
        [3., 1., 0.],
        [5., 7., 1.]], dtype=torch.float64)
tensor(0., dtype=torch.float64)
tensor(14., dtype=torch.float64)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

BruceWayne: <p>Hi K.Frank</p>
<p>Thanks for the respond, I have an idea that might be about to pass the symmetry information, just still trying to make it work in autograd if it’s possible, here’s my idea: suppose we define tensor X and tell autograd that it requires gradient info, we compute function value Y(X), and then we compute another function value Z(Y), and then finally compute the Jacobian dZ/dY instead of dZ/dX, is this possible?  If so I could do something like:</p>
<pre><code class="lang-auto">def Y(symmetric_tensor): return 0.5 * ( symmetric_tensor.T + symmetric_tensor )
</code></pre>
<p>and then Z(Y) would be the actual function, for example, the square of the input, to be taken jacobian of, this should provide symmetry information in the way that for example, dZ[1,2]/dY[21] = dZ(0.5(X[1,2]+X[2,1])) / d (0.5(X[1,2]+X[2,1]))   != 0</p>

BruceWayne: <pre><code class="lang-auto">import torch
X = torch.tensor([[1,3,5],[3,1,7],[5,7,1]],dtype=torch.double)

X.requires_grad_(True)

Y = 0.5 * (X+X.T)

def compute_Z(input):return torch.pow(input,2)
    
print(torch.autograd.functional.jacobian(compute_Z, Y))

</code></pre>
<p>Unfortunately this is still outputting the same result without the symmetry <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi Bruce!</p>
<aside class="quote no-group quote-modified" data-username="BruceWayne" data-post="4" data-topic="195596" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/d07c76/48.png" class="avatar"> BruceWayne:</div>
<blockquote>
<pre><code class="lang-auto">Y = 0.5 * (X+X.T)
</code></pre>
<p>Unfortunately this is still outputting the same result without the symmetry</p>
</blockquote>
</aside>
<p>Try moving your symmetrization step inside of <code>compute_Z()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.2
&gt;&gt;&gt;
&gt;&gt;&gt; X = torch.tensor ([[1, 3, 5], [3, 1, 7], [5, 7, 1]], dtype = torch.double)
&gt;&gt;&gt;
&gt;&gt;&gt; def compute_Z (input) :   # symmetrize input -- in a sense assumes that it is symmetric
...     Y = 0.5 * (input + input.T)
...     return torch.pow (Y, 2)
...
&gt;&gt;&gt; dZdX = torch.autograd.functional.jacobian (compute_Z, X)
&gt;&gt;&gt;
&gt;&gt;&gt; print (dZdX[1, 2, 1, 2])
tensor(7., dtype=torch.float64)
&gt;&gt;&gt; print (dZdX[1, 2, 2, 1])
tensor(7., dtype=torch.float64)
</code></pre>
<p>Does this work for you?</p>
<p>Best.</p>
<p>K. Frank</p>

BruceWayne: <p>Hi K.Frank again</p>
<p>Unfortunately, moving the symmetrization step into the function does not help, although the output shows symmetric, the values are not correct.<br>
This simpler test shows the problem, for any 3-by-3 tensor <code>X</code>, function <code>Y</code> is simple <code>X</code> itself plus the symmetrization:</p>
<pre><code class="lang-auto">X.requires_grad_(True)
def compute_Y(X):return 0.5 * (X+X.T)
print('dYdX = ', torch.autograd.functional.jacobian(compute_Y, compute_Y(X)))
</code></pre>
<p>The correct output with symmetric information passing down correctly would be a 4th order symmetric tensor with only 1 and zero components, however, the about code leaves 0.5 on the off-diagonal non-zero positions.</p>

KFrank: <p>Hi Bruce!</p>
<aside class="quote no-group" data-username="BruceWayne" data-post="6" data-topic="195596" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/d07c76/48.png" class="avatar"> BruceWayne:</div>
<blockquote>
<p>The correct output with symmetric information passing down correctly would be a 4th order symmetric tensor with only 1 and zero components</p>
</blockquote>
</aside>
<p>This depends on what you think the jacobian with respect to a symmetric<br>
matrix ought to be.</p>
<p>The core issue is that a 3x3 symmetric matrix has six degrees of freedom,<br>
namely the three diagonal elements plus the three off-diagonal elements<br>
(each of which occurs as two equal copies in the full 3x3 matrix).</p>
<p>In contrast, a general (that is, not necessarily symmetric) 3x3 matrix has<br>
nine degrees of freedom – all nine elements of the matrix are independent<br>
of one another.</p>
<p>Conceptually the gradient of a scalar function with respect to a 3x3<br>
symmetric matrix also consists of six (independent) values.  If you<br>
express that gradient as a full 3x3 matrix, you might be “over-counting”<br>
the off-diagonal elements of the gradient by a factor of two (depending<br>
on how you define things) because there are two copies of those<br>
off-diagonal elements.</p>
<p>You can view that factor of <code>0.5</code> as compensating for the over-counting.</p>
<p>If you don’t like that factor of <code>0.5</code>, you can compute the jacobian the first<br>
way I suggested and then symmetrize the result, or you can compute the<br>
jacobian the second way I suggested and multiply the off-diagonal elements<br>
by <code>2.0</code>.</p>
<p>You can use whatever definition you want for the the off-diagonal elements<br>
of the jacobian (as long as you use it consistently).</p>
<p>(My preference is to <em>admit</em> that a 3x3 symmetric matrix has only six<br>
independent degrees of freedom and represent it (and the gradient)<br>
as a triangular matrix with only six (non-zero) elements.)</p>
<p>Best.</p>
<p>K. Frank</p>