Use torch.svd with complex matrix on GPU

username: Yuchen_Mu
cooked: <p>Hi all, I am trying to calculate the SVD of some complex matrices. It worked fine on the CPU, but when it comes to the GPU, some errors happened, here is the snippet of my code and its corresponding output:</p>
<pre><code class="lang-auto">print("M_final is: ", M_final)
print(M_final.dtype)
print(M_final.device)
SVD_result = torch.svd(M_final)
print(SVD_result)

......Output:......

M_final is:  tensor([[ 0.2688+1.2463e-05j, -0.2223+1.0665e-01j],
        [-0.1943-1.3049e-01j,  0.0545-3.9980e-01j],
        [-0.5253-5.2017e-01j,  0.1036+3.6822e-01j],
        [-0.2718-4.8315e-01j, -0.2331-5.5959e-01j],
        [ 0.0798+1.1285e-01j, -0.2115-4.6674e-01j]], device='cuda:0')
torch.complex64
cuda:0
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-129-4a7caa855681&gt; in &lt;module&gt;
     30 print(M_final.dtype)
     31 print(M_final.device)
---&gt; 32 SVD_result = torch.svd(M_final)
     33 print(SVD_result)

RuntimeError: "svd_cuda" not implemented for 'ComplexFloat'
</code></pre>
<p>Does this mean that the <code>torch.svd</code> does not support the complex tensor on the GPU? I checked some posts and issues, it seemed that this functionality is actually supported on GPU: <a href="https://github.com/pytorch/pytorch/pull/42738" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/42738</a></p>
<p>Can anyone give some suggstions on this? Thanks in advance!</p>

username: KFrank
cooked: <p>Hello Yuchen!</p>
<aside class="quote no-group quote-modified" data-username="Yuchen_Mu" data-post="1" data-topic="105238" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/ea5d25/48.png" class="avatar"> Yuchen_Mu:</div>
<blockquote>
<p>Hi all, I am trying to calculate the SVD of some complex matrices. It worked fine on the CPU, but when it comes to the GPU, some errors happened,<br>
…<br>
Can anyone give some suggstions on this?</p>
</blockquote>
</aside>
<p>If you can tolerate the risks of living on the “bleeding edge,” it appears<br>
that the nightly build (version 1.8.0) now has complex svd on the gpu:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.8.0.dev20201203'
&gt;&gt;&gt; torch.svd (torch.randn ([2, 5], dtype = torch.cfloat, device = 'cuda:0'))
torch.return_types.svd(
U=tensor([[-0.9239+0.0000e+00j, -0.3826+1.1088e-08j],
        [ 0.0543+3.7870e-01j, -0.1311-9.1457e-01j]], device='cuda:0'),
S=tensor([2.3032, 1.6723], device='cuda:0'),
V=tensor([[ 0.3954-0.1435j, -0.2532+0.1638j],
        [ 0.0011+0.3421j,  0.1400-0.3463j],
        [ 0.0546-0.4828j, -0.3884-0.4571j],
        [ 0.1831+0.3384j, -0.4973-0.1605j],
        [-0.2001-0.5309j,  0.0373-0.3678j]], device='cuda:0'))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Yuchen_Mu
cooked: <p>Hi Frank!</p>
<p>Nice to see you here again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> and thank you very much! Let me try it and see what happens.</p>
<p>Regards,<br>
Yuchen</p>

username: Yuchen_Mu
cooked: <p>Hi Frank,</p>
<p>I tried the nightly version and yes, this version supported the (forward) computation of SVD for comlex matrix, but when I implemented a customized layer which included this complex matrix SVD cmputation, and put this layer inside my model, during the training time the following error occurred:</p>
<pre><code class="lang-auto">RuntimeError: svd does not support automatic differentiation for outputs with complex dtype.
</code></pre>
<p>It seems like 1.8 verson still does not support the aotugrad for SVD when complex matrix is used, any suggestions or should I wait Pytorch develop team to improve this functionality?</p>
<p>Regards,<br>
Yuchen</p>

username: KFrank
cooked: <p>Hello Yuchen!</p>
<aside class="quote no-group" data-username="Yuchen_Mu" data-post="4" data-topic="105238" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/ea5d25/48.png" class="avatar"> Yuchen_Mu:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: svd does not support automatic differentiation for outputs with complex dtype.
</code></pre>
<p>It seems like 1.8 verson still does not support the aotugrad for SVD when complex matrix is used, any suggestions</p>
</blockquote>
</aside>
<p>I don’t know of any quick fix for this.</p>
<aside class="quote no-group">
<blockquote>
<p>or should I wait Pytorch develop team to improve this functionality?</p>
</blockquote>
</aside>
<p>Yes, I think waiting is probably the best option.  Complex tensors are<br>
currently a work in progress in pytorch.</p>
<p>As a further note, I’ve been thinking about this some, and I don’t fully<br>
understand how autograd ought to work with complex tensor functions.<br>
If I can get my thoughts sorted out on this I will likely post something,<br>
but right now I am very confused by the whole thing.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Yuchen_Mu
cooked: <p>Ok I see it, thank you Frank!</p>