RuntimeError: "addcmul_cuda" not implemented for 'ComplexFloat'

EsonJohn: <p>Hello,</p>
<p>I have implemented a custom module that operates on complex tensors. The input, output and trainable weights in this module are with dtype=torch.complex64. When training this module with Adam optimizer on GPU device, I get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/MyName/.local/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/home/MyName/.local/lib/python3.6/site-packages/torch/optim/adam.py", line 119, in step
    group['eps']
  File "/home/MyName/.local/lib/python3.6/site-packages/torch/optim/functional.py", line 87, in adam
    exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)
RuntimeError: "addcmul_cuda" not implemented for 'ComplexFloat'
</code></pre>
<p>It seems that the <em><strong>torch.addcmul</strong></em> function could not be applied on complex tensors when operating on GPU. To validate this limitation of pytorch, I run the following code:</p>
<pre><code class="lang-auto">a=torch.randn(3,4,dtype=torch.complex64).cuda()
b=torch.randn(3,4,dtype=torch.complex64).cuda()
c=torch.randn(3,4,dtype=torch.complex64).cuda()
torch.addcmul(a,b,c)
</code></pre>
<p>As expected, the error message showed up:</p>
<pre><code class="lang-auto">RuntimeError: "addcmul_cuda" not implemented for 'ComplexFloat'
</code></pre>
<p>But when I run the above code on CPU, no error occurs.<br>
I also tested several other Pointwise Ops like <em><strong>add, mul, addcdiv</strong></em>, etc. It seems that only <em><strong>addcmul</strong></em> and <em><strong>addcdiv</strong></em> don’t support this feature (operate on complex tensors on GPU).</p>
<p>So, my question is, how could I modify the <em><strong>addcmul/addcdiv</strong></em> source code so that I can use Adam optimizer for my custom Module that operates on complex tensors? Would this issue going to be fixed in the comming pytorch versions?</p>
<p>Thanks in advance.</p>

ChrisLiu2: <p>Hi, I’m having the same issue. Have you found a solution yet?</p>

KFrank: <p>Hi Xulin and Zhang!</p>
<aside class="quote no-group" data-username="EsonJohn" data-post="1" data-topic="102645">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/esonjohn/48/31001_2.png" class="avatar"> EsonJohn:</div>
<blockquote>
<p>It seems that the <em><strong>torch.addcmul</strong></em> function could not be applied on complex tensors when operating on GPU.</p>
</blockquote>
</aside>
<p>Support for complex tensors in pytorch is a work in progress.  I find,<br>
just by trying, that <code>addcmul()</code> does not work with complex gpu tensors<br>
using pytorch version 1.6.0, but does work with a recent nightly build,<br>
version 1.8.0.dev20201203.</p>
<p>If you’re just experimenting with complex tensors, you might want to<br>
upgrade to a current (unstable) nightly build, but if you’re doing “real”<br>
work, it might make sense to avoid complex tensors until support for<br>
them becomes stable.</p>
<p>Best.</p>
<p>K. Frank</p>

ChrisLiu2: <p>Thanks for the help. This might be asking for too much, but do you know where I can find this type of information? Another part where I get stuck is complex tensor @, and I would want to check if this is available in preview as well.</p>

EsonJohn: <p>Hi Frank,</p>
<p>Thank you for the information. I will try it on my case latter.</p>
<p>Here is what I did to circumvent this issue:<br>
I converted complex128 tensor to a double channel float64 tensor, with its first channel stores the real part and the second channel stores the imag part. I also custom my own Linear and Conv2d layers, which take in the double channel tensor and perform Mul and Add operations according to the complex operation rules.</p>
<p>After these customization, the network has no complex calculation in its pipeline and could be optimized with traiditional optimizers.</p>
<p>Hope this information would be valuable to anyone in need.</p>
<p>Best,<br>
Yi Zhang</p>