The derivative for 'eig' is not implemented

username: 6fb4b64ff0a68bcbf8fa
cooked: <p>hello</p>
<p>my source</p>
<pre><code class="lang-auto"># ReEig Layer
def cal_rect_cov(features):
#     features=features.detach().cpu()
    result=[]
#     features=torch.tensor(features,requires_grad=True)
    for i in range(features.shape[0]):
        s_f,v_f=torch.eig(features[i], eigenvectors=True)
        s_f_convert=[]
        for i in range(len(s_f)):
            s_f_convert.append(s_f[i][0])
        s_f_convert = torch.stack(s_f_convert)
        s_f2=torch.clamp(s_f_convert,0.0001,10000)
        s_f2=torch.diag(s_f2)
        features_t=torch.matmul(torch.matmul(v_f,s_f2),v_f.t())
        result.append(features_t)
    result = torch.stack(result)
#     result=result.cuda()
    return result
</code></pre>
<p>loss calculate is successful</p>
<p>but</p>
<p>loss.backward()</p>
<p>in this source, error like this</p>
<p>the derivative for ‘eig’ is not implemented</p>
<p>i don’t know why this error occured.</p>
<p>plz help me</p>
<p>thank you</p>

username: KFrank
cooked: <p>Hi James!</p>
<aside class="quote no-group quote-modified" data-username="6fb4b64ff0a68bcbf8fa" data-post="1" data-topic="46962" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/6fb4b64ff0a68bcbf8fa/48/12755_2.png" class="avatar"> 6fb4b64ff0a68bcbf8fa:</div>
<blockquote>
<p>my source</p>
<pre><code class="lang-auto">        s_f,v_f=torch.eig(features[i], eigenvectors=True)
</code></pre>
<p>but</p>
<p>loss.backward()</p>
<p>in this source, error like this</p>
<p>the derivative for ‘eig’ is not implemented</p>
<p>i don’t know why this error occured.</p>
</blockquote>
</aside>
<p>Gradients (“backward pass”) are not implemented for<br>
<code>torch.eig()</code>.  If your <code>features</code> matrix is symmetric<br>
(or can naturally be made symmetric), you can use<br>
<code>torch.symeig()</code>, otherwise you will have to rethink<br>
your loss function.</p>
<p>From the documentation for <a href="https://pytorch.org/docs/stable/torch.html#torch.eig" rel="noopener nofollow ugc">torch.eig()</a>:</p>
<pre><code class="lang-plaintext">Since eigenvalues and eigenvectors might be
complex, backward pass is supported only for
torch.symeig()
</code></pre>
<p>Also note, from <a href="https://pytorch.org/docs/stable/torch.html#torch.symeig" rel="noopener nofollow ugc">torch.symeig()</a>:</p>
<pre><code class="lang-plaintext">Extra care needs to be taken when backward through
outputs. Such operation is really only stable when
all eigenvalues are distinct. Otherwise, NaN can
appear as the gradients are not properly defined.
</code></pre>
<p>Let me just say that there is nothing fundamentally improper<br>
about using eigenvalues (or eigenvectors) as ingredients in<br>
your loss function, but eigenvalues can be a little tricky, both<br>
in terms of how they depend mathematically on your inputs,<br>
and sometimes the numerical stability of the algorithms used<br>
to calculate them.  So you should think through carefully how<br>
and why you’re using them.</p>
<p>Cheers!</p>
<p>K. Frank</p>

username: 6fb4b64ff0a68bcbf8fa
cooked: <p>thank you for your answer.</p>
<p>I solved this error but</p>
<p>you already worry about gradient print none value.</p>
<p>I check my gradient eigenvalue,eigenvector(s_f.grad,v_f.grad).</p>
<p>only print none value.</p>
<p>so I consider more backward about symeig.</p>
<p>I studied about "Covariance Pooling for Facial Expression Recognition<br>
"</p>
<p>this paper offered tensorflow code.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/d-acharya/CovPoolFER/blob/master/conv_feature_pooling/src/models/covpoolnet.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/d-acharya/CovPoolFER/blob/master/conv_feature_pooling/src/models/covpoolnet.py" target="_blank" rel="nofollow noopener">d-acharya/CovPoolFER/blob/master/conv_feature_pooling/src/models/covpoolnet.py</a></h4>
<pre><code class="lang-py">from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import tensorflow as tf
import tensorflow.contrib.slim as slim

def inference(images, keep_probability, phase_train=True, bottleneck_layer_size=7, weight_decay=0.0, reuse=None, batch_size=128):
    batch_norm_params={
        'decay':0.995,
        'epsilon':0.001,
        'updates_collections':None,
        'variables_collections':[tf.GraphKeys.TRAINABLE_VARIABLES]
    }
    with slim.arg_scope([slim.conv2d, slim.fully_connected],
        weights_initializer=tf.truncated_normal_initializer(stddev=0.1),
        weights_regularizer=slim.l2_regularizer(weight_decay),
        normalizer_fn=slim.batch_norm,
        normalizer_params=batch_norm_params):
        return network(images, is_training=phase_train,dropout_keep_prob=keep_probability,bottleneck_layer_size=bottleneck_layer_size, batch_size)
</code></pre>

  This file has been truncated. <a href="https://github.com/d-acharya/CovPoolFER/blob/master/conv_feature_pooling/src/models/covpoolnet.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>every thing changed pytorch.</p>
<p>but just problemed om symeig.</p>
<p>any other idea about symeig do not get gradient?</p>
<p>thank you for your answer again.</p>
<p>-James</p>

username: KFrank
cooked: <p>Hi James!</p>
<p>First a comment:</p>
<aside class="quote no-group quote-modified" data-username="6fb4b64ff0a68bcbf8fa" data-post="3" data-topic="46962" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/6fb4b64ff0a68bcbf8fa/48/12755_2.png" class="avatar"> 6fb4b64ff0a68bcbf8fa:</div>
<blockquote>
<p>I studied about “Covariance Pooling for Facial Expression Recognition”</p>
<p>this paper offered tensorflow code.<br>
…<br>
every thing changed pytorch.</p>
</blockquote>
</aside>
<p>It sounds like you are trying to translate the “Covariance Pooling”<br>
algorithm from tensorflow to pytorch.  This might well involve<br>
some amount of work, but I would expect it to be doable.</p>
<p>And a question:</p>
<aside class="quote no-group">
<blockquote>
<p>I check my gradient eigenvalue,eigenvector(s_f.grad,v_f.grad).</p>
<p>only print none value.</p>
<p>so I consider more backward about symeig.<br>
…<br>
but just problemed om symeig.</p>
<p>any other idea about symeig do not get gradient?</p>
</blockquote>
</aside>
<p>It sounds like you are saying that <code>symeig</code> is giving you “none” for<br>
the value of its gradient.  Is this the issue you are asking about?<br>
(I should note that I haven’t ever used <code>symeig</code>.)</p>
<p>Could you try to isolate this problem and post a small, complete,<br>
runnable pytorch script where you evaluate <code>symeig</code> on some<br>
inputs, call <code>backward()</code>, and then print out the gradients you<br>
get?</p>
<p>If we start with something simple like that, we can try to analyze<br>
your gradient issue.</p>
<p>Best.</p>
<p>K. Frank</p>

username: 6fb4b64ff0a68bcbf8fa
cooked: <p>Thank you for your reply.</p>
<pre><code class="lang-auto">I check my gradient eigenvalue,eigenvector(s_f.grad,v_f.grad).

only print none value.
</code></pre>
<p>this mean that during training stage,</p>
<p>I printed eigenvalue’s grad and eigenvector’s grad.</p>
<p>in training stage, they printed None value.</p>

username: KFrank
cooked: <p>Hello James!</p>
<aside class="quote no-group" data-username="6fb4b64ff0a68bcbf8fa" data-post="5" data-topic="46962" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/6fb4b64ff0a68bcbf8fa/48/12755_2.png" class="avatar"> 6fb4b64ff0a68bcbf8fa:</div>
<blockquote>
<p>Thank you for your reply.</p>
<pre><code class="lang-auto">I check my gradient eigenvalue,eigenvector(s_f.grad,v_f.grad).

only print none value.
</code></pre>
<p>this mean that during training stage,</p>
<p>I printed eigenvalue’s grad and eigenvector’s grad.</p>
<p>in training stage, they printed None value.</p>
</blockquote>
</aside>
<p>Well, if you’re getting “none” gradients during training, there’s<br>
almost certainly a bug somewhere, more likely in your code,<br>
but possibly, though less likely, in pytorch.</p>
<p>If you don’t find the bug fairly quickly just looking through your<br>
code, then more orderly debugging is called for.  A common<br>
approach is to break thing up into pieces to try and isolate the<br>
bug.</p>
<p>As I suggested in my previous post, you might try just seeing<br>
if you can get a gradient out of <code>symeig</code>.  My suggestion would<br>
be that you post a short, complete, runnable pytorch script<br>
that pumps a <code>tensor</code> (with <code>requires_grad</code> set) through<br>
<code>symeig</code> and somehow calculates a scalar (single number)<br>
from that.  Then call <code>backward()</code> on that scalar and see if<br>
your input <code>tensor</code> has good gradients or just “none”.</p>
<p>If you post the script and your results, forum participants can<br>
not only look at it, but run it for themselves, and maybe they<br>
will see something or have ideas about what is going on.</p>
<p>Good luck.</p>
<p>K. Frank</p>