How to use complex eigenvalues?

ksteensig: <p>I have a nonsymmetric real matrix that I want to calculate the eigenvalues and matrices for. I use the torch.eig() method which works.<br>
But my eigenvalues are complex and I am unsure of how to handle this.<br>
I am using the following to go from eigenvalues/vectors and back.</p>
<pre><code class="lang-python">X = torch.rand(20,20)
D,T = torch.eig(X, eigenvectors=True)
T.mm(torch.diag(D)).mm(T.t())
</code></pre>
<p>But because D consists of complex eigenvalues, and in PyTorch a complex value is a vector of two values, I am unable to calculate the original matrix again using matrix multiplication. How will I be able to do that? I know <code>torch.diag(D)</code> results in a vector because <code>D</code> is a matrix, but how are you supposed to multiple a complex matrix in PyTorch?</p>

KFrank: <p>Hello Kasper!</p>
<aside class="quote no-group" data-username="ksteensig" data-post="1" data-topic="71983" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ksteensig/48/21251_2.png" class="avatar"> ksteensig:</div>
<blockquote>
<p>But because D consists of complex eigenvalues, and in PyTorch a complex value is a vector of two values, I am unable to calculate the original matrix again using matrix multiplication.</p>
</blockquote>
</aside>
<p>Pytorch does not currently support complex tensor operations (to<br>
the best of my knowledge).</p>
<p>First, what is your actual use case?  There may be some other way<br>
to accomplish your goal.</p>
<p>Second, for work on complex tensors in pytorch, please see this<br>
post:</p>
<aside class="quote" data-post="1" data-topic="60920">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/97f17d/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-with-complex-tensors/60920">Pytorch with complex tensors</a> 
  </div>
  <blockquote>
    Is there a solution for dealing with complex tensors yet? Is there a similar function to numpy.angle in PyTorch? 
thanks!
  </blockquote>
</aside>

<p>and the linked github issue:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/755">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/755" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/755" target="_blank" rel="noopener nofollow ugc">Integrating complex tensors</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-02-15" data-time="23:06:57" data-timezone="UTC">11:06PM - 15 Feb 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2022-02-08" data-time="16:25:56" data-timezone="UTC">04:25PM - 08 Feb 22 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/PhilippPelz" target="_blank" rel="noopener nofollow ugc">
          <img alt="PhilippPelz" src="https://avatars.githubusercontent.com/u/1299153?v=4" class="onebox-avatar-inline" width="20" height="20">
          PhilippPelz
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          feature
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: complex
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">New description from @ezyang:

Work is in progress at https://github.com/Roger<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">-luo/pytorch-complex

## Organizational principles

* Complex tensor support is important to PyTorch, and we will accept patches to core which add small amounts of code to make adding complex support.
* Adding complex involves writing a lot of new kernels and code: we'd like this code to initially live out of repo, so it is easier for people to iterate quickly on them without having to go through the PyTorch main code review process. We will *NOT* commit to reviewing large new kernels in the short term, but eventually we would like all the kernels to come back to PyTorch.
* The external library will be buildable separately from PyTorch, so you will be able to maintain it as a separate repository without having to merge with PyTorch (and deal with loads of merge conflicts).
    * PyTorch may occasionally make breaking changes in C++ API; if you bring these to our attention we will do our utmost to help solve these problems.
* The hooks needed for this will NOT ship with PyTorch 1.0, but they will ship with a released version of PyTorch in the not too distant future.

## How will I work on complex kernels?

Here is what the workflow will look like in the steady state.

**PyTorch will natively contain APIs for referring to the complex dtype, but they won't do anything by default.** PyTorch defines torch.complex64 and torch.complex128 referring to complex tensors. However, if you try to construct a tensor this way, by default, PyTorch will error:

```
&gt;&gt;&gt; torch.zeros({2,2}, dtype=torch.complex64)
RuntimeError: complex64 not supported by PyTorch
```

@ezyang provided a patch which adds these dtypes to PyTorch. https://github.com/pytorch/pytorch/pull/11173

In the mid-term, we will merge support for basic functionality (like allocating a tensor of zeros) to be supported by PyTorch natively. A reasonable proxy for what support is “basic” is PyTorch's native support for CPU half tensors (which are extremely impoverished).

**PyTorch publishes an interface for registering an implementation of complex tensors.** The implementation inherits from the TypeDefault class (https://github.com/pytorch/pytorch/pull/11013) and will override methods on this class to define implementations of functions for which we have complex implementations. It will look something like this:

```
struct CPUComplexFloatType final : public TypeDefault {
  virtual Tensor add(const Tensor &amp; self, const Tensor &amp; other, Scalar alpha=1) const override {
    // Your implementation of add for complex tensors
  }
  // ...
}
```

This class will override exactly the types which are supported for complex; all other implementations are provided by TypeDefault and will error by default.

There will be a canonical listing of methods supported on Type (the overall interface) as an autogenerated file that is checked into the PyTorch source repository; we'll communicate API changes by diffs to this file. In general, the methods are in one-to-one correspondence with their corresponding names in the PyTorch frontend.

In general, when you use an operation which you haven't implemented yet, 

**WARNING:** We intend to refactor Type away into a new system that also supports open registration of new operations (this obviously doesn't work if you have a single superclass that defines all the methods you might possibly want to support). Thus, try not to get too tied to the particular implementation strategy of writing Type as a subclass.

**To publish new, complex only operations, you will use the C++ extension API.** The C++ extension API is documented at https://pytorch.org/tutorials/advanced/cpp_extension.html Essentially, you can write a C++ function like:

```
at::Tensor imag(at::Tensor z) {
  ...
}
```

And then the C++ extension API will generate a Python binding so that you invoke this function from Python.

**Some operations will be “easy” to integrate into PyTorch as it exists today.** For example, for implementation of binary operations, it probably makes more sense to extend add_kernel in BinaryOpsKernel.cpp so that it dispatches over complex types (and then you get it for free, because std::complex implements addition). As long as these patches are small and self-contained, we promise to merge them on a timely basis.

It should ALWAYS be possible to unblock, by just writing an override on Type instead of using existing infrastructure, and doing liberal copy pasting. But let's avoid it when it's easy!

**Autograd.** As long as you're working on operations which already have derivative formulas defined for them, you will “automatically” get autograd support, as long as you implement complex support for all the constituent functions which are invoked in the backwards implementation from derivatives.yaml.

In some cases, we may need to adjust autograd formulas so that they work for complex numbers; e.g., the gradient of 'abs' isn't 'grad . self.sign()'. In these cases, all we need to do is upstream fix of changing the autograd formula of 'abs' to 'abs_backward', which is a function that can be overridden.

For general complex valued back propagation, there are some references:

1. *Akira’s “Complex Valued Neural Networks”.*
2. https://giggleliu.github.io/2018/02/01/complex_bp.html

Generally, we won't need to modify the autograd since in most cases we only calculate the derivatives of a real-valued function (the loss). 

## Work plan

Many of the necessary pieces are in place today, but they are not put together in an end-to-end way. Here is what needs to be done.

- [X] Codemod TH to not ifdef real https://github.com/pytorch/pytorch/pull/11163
- [X] Built-in support for torch.complex64 and torch.complex128 dtypes. https://github.com/pytorch/pytorch/pull/11173
- [X] An interface for registering CPUComplexType, etc., so that this implementation is invoked when you request a complex tensor with dtype=torch.complex64 or do an operation on complex tensors. 
- [X] Land https://github.com/pytorch/pytorch/pull/11013
- [X] An end-to-end example, including working build system, of a separately compileable C++ program that links against libtorch and uses the aforementioned interface to implement complex tensor allocation.

Short term integration plan. These operations are “easy” to implement, and so we should mainline them in PyTorch as soon as possible.

- [X] Basic tensor factories: torch.empty, torch.zeros, torch.ones
- [ ] CPU binary operations: add, sub, mul, div #11641
- [ ] FFT
- [ ] ???

Kernel implementation:

TODO: Generate a list based on https://github.com/Roger-luo/TH/blob/master/ChangeLog.md

Other complex related tasks:

- [ ] Figure out the type promotion rules for complex tensors, and implement it in promoteTypes #11641

## Historical issue content

Original comment from @PhilippPelz 

I was wondering if there is interest in incorporating complex tensors into pytorch.
For CPU support there is ztorch and I have written z-cutorch ( https://github.com/PhilippPelz/z-cutorch ) a while ago. It is a fork off cutorch before the refactoring for CudaHalfTensor (don't have the hardware yet). 
If it's not too much work, I would like to slowly integrate it with pytorch. I am using matplotlib for plotting via fb.ptyhon and it turns out a huge pain every time I reinstall my system (compiling all the dependencies), plus it seems pytorch will work under Windows soon, which one of my experiment PCs runs on.
I would also need complex gradients, so I would sooner or later touch autograd as well. 
While tf supports complex tensors per se, it seems many ops don't support it yet (https://github.com/tensorflow/tensorflow/issues/2255), plus it seems a bit heavyweight for my purposes.

Maybe someone could say a few words how and where to start with this, if it's a welcome idea.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Last, for a number of operations, including matrix multiplication, it is<br>
easy enough to build your own complex tensor operations out of<br>
real tensor operations.</p>
<p>You can write <code>cm = rm + I * im</code>, where <code>cm</code> is your complex matrix,<br>
<code>rm</code> and <code>im</code> are real matrices, and <code>I</code> is the notional <code>sqrt (-1)</code> (the<br>
“imaginary unit”).  You can then expand out the product of two complex<br>
matrices in terms of these real matrices, and collect terms to get the<br>
real and imaginary parts of your product matrix.  (In your example of<br>
reconstructing the real matrix that you diagonalized you should find<br>
that the imaginary part of your final result is zero to within floating-point<br>
precision.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

ksteensig: <p>I am using the complex eigenvalues for a low-rank approximation of the singular values for my matrix.<br>
The algorithm I am using has an intermediate step of calculating the eigenvalues/vectors of a much smaller matrix.<br>
In this step I actually have to take the square root of the diagonal matrix containing the eigenvalues.</p>
<p>But thank you very much, I will see if I can expand this into something that can be calculated using only real valued matrices.</p>
<p>EDIT: Later on I have figured out I am in fact calculating the eigenvalues for a PSD matrix, so the complex values are small enough to be ignored.</p>