Autograd on complex numbers

ztan: <p>Dear PyTorch Developers,</p>
<p>I started to play with the autograd function in PyTorch, and wrote the following simple example:</p>
<pre><code class="lang-python">import numpy as np
import torch

dat = np.array([[1. + 2.j, 3. + 4.j], [5. + 6.j, 7. + 8.j]], dtype=np.complex)
x = torch.tensor(dat, requires_grad=True)

y = 2.0 * x**2

extern_grad = torch.tensor(np.ones_like(dat))
y.backward(gradient=extern_grad)

print(4 * dat)
print(x.grad)
</code></pre>
<p>The print results show that the pre-computed gradient (4 * dat) is the conjugate of the autograd (x.grad). However, I expect that they should be identical. Does the way how I used autograd contain some errors?</p>
<p>With best regards,</p>

cahity: <p>In PyTorch 1.6.0, I get identical results for both gradient calculations. In latest PyTorch version (1.10.0)  though, I get what you got. It is because autograd calculates Wirtinger derivatives.</p>
<p>More info at: <a href="https://pytorch.org/docs/stable/notes/autograd.html#autograd-for-complex-numbers" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/autograd.html#autograd-for-complex-numbers</a></p>

KFrank: <p>Hi Zhengguo!</p>
<aside class="quote no-group" data-username="ztan" data-post="1" data-topic="144687" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ztan/48/46890_2.png" class="avatar"> ztan:</div>
<blockquote>
<p>The print results show that the pre-computed gradient (4 * dat) is the conjugate of the autograd (x.grad).</p>
</blockquote>
</aside>
<p>[Edit:  The link that y cahit posted looks like a good explanation.]</p>
<p>I believe that this behavior is by design.</p>
<p>I’m foggy on the details, but I think that this choice is driven by how<br>
you want gradient descent to work with gradients of (real-valued)<br>
losses with respect to complex parameters.</p>
<p>The best (but imperfect) discussion I know of is this github issue:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/41857">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/41857" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/41857" target="_blank" rel="noopener nofollow ugc">Follow the JAX or Tensorflow convention for meaning of grad() with complex inputs</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-07-22" data-time="18:29:42" data-timezone="UTC">06:29PM - 22 Jul 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2021-01-15" data-time="20:24:44" data-timezone="UTC">08:24PM - 15 Jan 21 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/ezyang" target="_blank" rel="noopener nofollow ugc">
          <img alt="ezyang" src="https://avatars.githubusercontent.com/u/13564?v=4" class="onebox-avatar-inline" width="20" height="20">
          ezyang
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          high priority
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: autograd
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: complex
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">cc #755 

JAX and Tensorflow disagree about whether or not the grad of a compl<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ex tensor should be conjugated or not. Here is an easy way to see the difference:

```
from jax import grad

def f(z):
  return z * z

z = 0.1j

print(grad(f, holomorphic=True)(z))
```

gives 0.2j

However

```
&gt;&gt;&gt; x = tf.Variable(0. + 0.j)
&gt;&gt;&gt; sess.run(tf.gradients(x*x, x), feed_dict={x:0.1j})
[-0.20000000000000001j]
```

source: https://github.com/tensorflow/tensorflow/issues/3348

PyTorch also has to decide which side of the field it will come down on. Right now on master it implements JAX. From reading the issue, here is my understanding of the pros and cons:

In favor of TF:
* The gradient is the correct direction for doing gradient descent. This means you can use a "stock" optimizer (one that was written with only real parameters in mind) without any changes to the optimizer. In contrast, to do gradient descent with the JAX definition, you have to remember to conjugate first.

In favor of JAX:
* When the TF definition is implemented directly in the gradient formulas (as opposed to doing a single post facto conjugation), you end up with ugly gradient formulas. For example, take a look at https://github.com/tensorflow/tensorflow/blob/70fd0a4436e3b49139653dc5b85d1c7df23f403d/tensorflow/python/ops/math_grad.py#L453 where TF has to explicitly conjugate the input. With the JAX style definition, you can mostly reuse your real gradient formulas.
* The TF definition is less efficient, since you're doing extra conjugations in the gradient formulas. Even if you write the derivative formulas in the JAX way and then conjugate before plopping the gradient in x.grad, you lose the opportunity to do a fused conjugate-add for the optimizer update.

Gradient formulas look ugly seems like a clear reason to prefer JAX style. Posting this for other opinions.

cc @ezyang @gchanan @zou3519 @SsnL @albanD @gqchen @anjali411 @dylanbespalko @vincentqb</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Perhaps <a class="mention" href="/u/alband">@albanD</a> has some updated information or perhaps a link<br>
to more expository documentation.</p>
<p>Best.</p>
<p>K. Frank</p>

albanD: <p>That discussion is indeed pretty accurate.<br>
And indeed the main motivation is for the optimizer to be able to be re-used with <code>p = p - lr * grad</code> where lr is real will move you in the right direction.</p>