Autograd for complex-valued neural networks

Anirudh_Sikdar: <p>Hi,<br>
I have a doubt for autograd for complex-valued neural networks(<a href="https://pytorch.org/docs/stable/notes/autograd.html" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 1.11.0 documentation</a>).It seems that autograd works when differentiating complex-valued tensors. But does it also work for layers in a neural network?<br>
So lets say I define a F.Linear layer, can we define the weights to be complex-valued and will the updates to this layer follow complex-valued backpropogation? Also, can the same thing be done for nn.Conv2d layer?</p>

KFrank: <p>Hi Anirudh!</p>
<aside class="quote no-group" data-username="Anirudh_Sikdar" data-post="1" data-topic="151160" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anirudh_sikdar/48/49114_2.png" class="avatar"> Anirudh_Sikdar:</div>
<blockquote>
<p>So lets say I define a F.Linear layer, can we define the weights to be complex-valued and will the updates to this layer follow complex-valued backpropogation?</p>
</blockquote>
</aside>
<p>Yes, autograd performs complex backpropagation through (most, if not<br>
all) pytorch layers and with respect to those layers’ parameters.</p>
<p>Consider this <code>Linear</code> example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.11.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; lin = torch.nn.Linear (2, 2, bias = False, dtype = torch.complex64)
&gt;&gt;&gt; lin.weight
Parameter containing:
tensor([[-0.1474+0.5967j,  0.3660-0.1681j],
        [-0.6700-0.1989j,  0.4149+0.3975j]], requires_grad=True)
&gt;&gt;&gt; t = torch.randn (5, 2, dtype = torch.complex64)
&gt;&gt;&gt; t
tensor([[-0.5672+0.5518j, -0.8078+0.3621j],
        [ 0.0075+0.4234j, -0.4115-0.4382j],
        [-1.5592+0.4851j,  0.6728-0.1796j],
        [ 0.4976+0.4027j, -0.9824-0.7075j],
        [-0.1651-0.3127j, -0.6328-0.4980j]])
&gt;&gt;&gt; lin (t).norm().backward()
&gt;&gt;&gt; lin.weight.grad
tensor([[-0.3748+1.0385j,  0.6620-0.4970j],
        [-1.2673-0.4408j,  0.8804+0.7391j]])
</code></pre>
<p>Note, the implementation of complex gradients has built into it, in some<br>
sense, the notion that you are calculating the gradient of a real-valued<br>
loss function.</p>
<p>If you calculate the gradient of a complex “loss” function, you might not<br>
get the results you expect (depending upon your expectations).</p>
<p>Pytorch’s support for complex manipulations is still a work in progress, but<br>
most (if not all) of the basics have been implemented and work correctly.</p>
<p>Best.</p>
<p>K. Frank</p>

Anirudh_Sikdar: <p>Hi K.Frank,<br>
Thanks for replying.I was trying to do the same for convolutional layer.<br>
The following code is a normal convolution operation.</p>
<blockquote>
<p>m = torch.nn.Conv2d(16, 33, (3, 3))<br>
input = torch.randn(20, 16, 50, 50)<br>
output = m(input)<br>
print(output)</p>
</blockquote>
<p>But when I do it for complex-convolution,</p>
<blockquote>
<p>m = torch.nn.Conv2d(16, 33, (3, 3),dtype=torch.complex64)<br>
input = torch.randn(20, 16, 50, 50,dtype=torch.complex64)<br>
output = m(input)<br>
print(output)</p>
</blockquote>
<p>It gives me the following error,</p>
<blockquote>
<p>RuntimeError: “slow_conv2d_cpu” not implemented for ‘ComplexFloat’.</p>
</blockquote>

KFrank: <p>Hi Anirudh!</p>
<aside class="quote no-group quote-modified" data-username="Anirudh_Sikdar" data-post="3" data-topic="151160" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anirudh_sikdar/48/49114_2.png" class="avatar"> Anirudh_Sikdar:</div>
<blockquote>
<p>It gives me the following error,</p>
<blockquote>
<p>RuntimeError: “slow_conv2d_cpu” not implemented for ‘ComplexFloat’.</p>
</blockquote>
</blockquote>
</aside>
<p>You’re right, <code>Conv2d</code> does not yet have complex support in the latest<br>
stable release (1.11.0) – that’s my fault for not double-checking.</p>
<p>(As I said, pytorch’s complex support is still a work in progress.)</p>
<p>However, <code>Conv2d</code> does have complex support in the latest nightly build<br>
(a 1.12 version):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0.dev20220510'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; conv = torch.nn.Conv2d (1, 1, 3, dtype = torch.complex64)
&gt;&gt;&gt; conv (torch.randn (1, 1, 3, 3, dtype = torch.complex64))
tensor([[[[0.1691-0.2932j]]]], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Anirudh_Sikdar: <p>Hi,<br>
Can you please tell me how the complex-valued weights are initialized for complex-convolution and complex-valued linear layers?<br>
My assumption is that both the real and imaginary components are independently initialized using the Xavier uniform initialization.</p>