How might autograd differentiate complex functions that are not complex-differentiable?

KFrank: <p>Hello Forum!</p>
<p>There are useful functions of a complex variable, <code>z</code>, that are not<br>
“complex-differentiable,” that is, they are not <em>analytic</em> functions.<br>
Two examples would be <code>z.conj()</code> and <code>|z|**2 = z * z.conj()</code>.</p>
<p>Note that these functions are differentiable when viewed as real<br>
functions of two real variables, <code>x</code> and <code>y</code>, where <code>z = x + yj</code>.</p>
<p>I think that when a function is analytic, autograd should return<br>
its conventional complex derivative.  However, we would like<br>
autograd also to do something “useful” for non-analytic functions<br>
that are differentiable when understood as real functions.</p>
<p>For example, it would be nice if gradient descent would work for<br>
minimizing <code>|z - z0|**2 = (z - z0) * (z - z0).conj()</code> with<br>
respect to <code>z</code> (which takes on its minimum value of <code>0</code> when <code>z = z0</code>).</p>
<p>I don’t have a well-thought-out proposal for how to do this.  But I<br>
came across this exposition that discusses some of the core issues:</p>
<p><a href="http://dsp.ucsd.edu/~kreutz/PEI-05%20Support%20Files/Lecture%20Supplement%203%20on%20the%20Complex%20Derivative%20v1.3c%20F05%20.pdf" rel="noopener nofollow ugc">The Complex Gradient Operator and the CR-Calculus</a></p>
<p>One facet of this issue is illustrated in this post where naive gradient<br>
descent is shown to fail for <code>|z|</code> (for complex <code>z</code>) using autograd in<br>
version 1.6.0:</p>
<aside class="quote quote-modified" data-post="3" data-topic="104547">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-implement-a-custom-loss-function-which-include-frobenius-norm/104547/3">How to implement a custom loss function which include frobenius norm?</a> 
  </div>
  <blockquote>
    Hello Yuchen! 

Complex tensors are still a work in progress in pytorch, with more 
functionality (and more functionality that is actually correct) being 
added in successive versions. 
Note that as of version 1.6.0, I don’t believe that the pytorch optimizers 
accept complex Parameters, so to use pytorch’s complex machinery, 
you will have to either use real Parameters that you combine into 
complex tensors or write your own complex-aware optimizer. 
All in all, depending on what you are doing,…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

albanD: <p>Hi Frank,</p>
<p>We actually had a very long discussion about this exact question and what to do.<br>
It is also quite fun to see that jax and Tensorflow currently define two different gradients for these non-analytical functions!<br>
You can see the whole discussion here: <a href="https://github.com/pytorch/pytorch/issues/41857" class="inline-onebox">Follow the JAX or Tensorflow convention for meaning of grad() with complex inputs · Issue #41857 · pytorch/pytorch · GitHub</a><br>
Also you can find in our doc a fairly extensive summary of the result of that discussion: <a href="https://pytorch.org/docs/master/notes/autograd.html#autograd-for-complex-numbers" class="inline-onebox">Autograd mechanics — PyTorch master documentation</a></p>
<p>Happy to discuss it more if you’re interested!</p>