Complex valued neural network

username: learner47
cooked: <p>Hi,</p>
<p>I am trying to used complex valued data as input to a test neural network. From the release notes, PyTorch 1.8.0 is said to support complex autograd. My code is as follows.</p>
<pre><code class="lang-auto">import torch
from torch import nn, optim


class ComplexTest(nn.Module):
    def __init__(self):
        super(ComplexTest, self).__init__()
        self.fc1 = nn.Linear(10, 20)
        self.fc2 = nn.Linear(20, 10)
        self.relu = nn.ReLU()

    def forward(self, inputs):
        return self.fc2(self.relu(self.fc1(inputs)))


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
complex_test = ComplexTest().to(device)
complex_test.train()

opt = optim.Adam(complex_test.parameters())

mse_loss = nn.MSELoss()

for _ in range(100):
    opt.zero_grad()

    inp = torch.randn((1000, 10), dtype=torch.cfloat).to(device)
    op = complex_test(inp)

    loss = mse_loss(op, inp)
    loss.backward()
    opt.step()
    print(loss.item())
</code></pre>
<p>But this gives an error</p>
<blockquote>
<p>expected scalar type Float but found ComplexFloat</p>
</blockquote>
<p>Is this not supported or did I read the documentation wrong? Thanks!</p>

username: omarfoq
cooked: <p>Hi, when you create <code>nn.Linear</code> it is by default initialized with Float, you may be need to cast the network weights to be complex after you create it.</p>

username: learner47
cooked: <p><a class="mention" href="/u/omarfoq">@omarfoq</a>, thank you for your response! I tried adding <code>to(torch.cfloat)</code> like you mentioned and my code looks as below.</p>
<pre><code class="lang-auto">import torch
from torch import nn, optim


class ComplexTest(nn.Module):
    def __init__(self):
        super(ComplexTest, self).__init__()
        self.fc1 = nn.Linear(10, 20).to(torch.cfloat)
        self.fc2 = nn.Linear(20, 10).to(torch.cfloat)
        self.relu = nn.ReLU()

    def forward(self, inputs):
        return self.fc2(self.relu(self.fc1(inputs)))


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
complex_test = ComplexTest().to(device)
complex_test.train()

opt = optim.Adam(complex_test.parameters())

mse_loss = nn.MSELoss()

for _ in range(100):
    inp = torch.randn((1000, 10), dtype=torch.cfloat)
    op = complex_test(inp)

    loss = mse_loss(op, inp)
    loss.backward()
    opt.step()
    print(loss.item())
</code></pre>
<p>Now the layers do not have problem with the complex valued input. But I could not figure out how to do this for activation function. The following error thereby pops up.</p>
<blockquote>
<p>“threshold_cpu” not implemented for ‘ComplexFloat’</p>
</blockquote>
<p>How can I proceed?</p>

username: omarfoq
cooked: <p>Hello,</p>
<p>I think ReLU activation doesn’t work for complex numbers</p>

username: learner47
cooked: <p>Hello again! So the dummy setup that I have mentioned is not runnable under any circumstance with pytorch 1.8.1? Was really looking forward to it.</p>

username: omarfoq
cooked: <p>Hello,</p>
<p>ReLU is in fact a function based on  comparing a real number with 0, and as there is no natural order for complex numbers, doing the max between a complex number and zero is not well defined, and so is ReLU for complex numbers.</p>
<p>Depending on what problematic you are working, I think you can define your own activations or use activations that are adapted to complex numbers (this includes all holomorphic functions)</p>

username: arpane4c5
cooked: <p>Hello,<br>
For those, who are searching complex valued NN/CNN implementations, the following library can be of help:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/wavefrontshaping/complexPyTorch">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/wavefrontshaping/complexPyTorch" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/99cf87898eb913b31c33dc487c0c0ef38d0e1560f1a9df9ad778cdf8a451cc65/wavefrontshaping/complexPyTorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/wavefrontshaping/complexPyTorch" target="_blank" rel="noopener nofollow ugc">GitHub - wavefrontshaping/complexPyTorch: A high-level toolbox for using...</a></h3>

  <p>A high-level toolbox for using complex valued neural networks in PyTorch  - GitHub - wavefrontshaping/complexPyTorch: A high-level toolbox for using complex valued neural networks in PyTorch</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
It implements the complex value based activations and layers. The Relu function for complex values has been implemented as CRelu.</p>