Torch.svd is slow in GPU compared to CPU

username: Sabra_Ossen
cooked: <p>In the following graph, you can see the execution time in seconds taken for Torch.svd in both CPU and GPU. As you can see the time taken for GPU is larger than that time taken for CPU and keeps increasing as we increase the number of records. Please note that the GPU memory has a size of 12 GB and the maximum number of records that can be run with svd is 3.25 million.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0e51cb5abb53e6d4d6e238f9a76d3122cd30be54.png" data-download-href="https://discuss.pytorch.org/uploads/default/0e51cb5abb53e6d4d6e238f9a76d3122cd30be54" title="chart (4)"><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0e51cb5abb53e6d4d6e238f9a76d3122cd30be54.png" alt="chart (4)" data-base62-sha1="22FVxVeQtXt3hr1FIXya14GMtJq" width="690" height="426" data-dominant-color="FBFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">chart (4)</span><span class="informations">997×616 7.59 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Following is the code I used.</p>
<pre><code>import torch
from datetime import datetime
import sys

n_records = sys.argv[1];
x = torch.zeros(n_records, 300).cuda()
s = datetime.now()
u, _, _ = torch.svd(x, some=True)
e = datetime.now()
del x
el = (e-s).total_seconds()
print(el)
</code></pre>
<p>Following is the output from NVIDIA visual profiler for 1000 and 1 million records respectively. As you can see after the ‘kernelPointWiseApply2’ cuda kernel invocation there is a certain amount of idle time that is introduced. This idle time between the first and second kernel functions in the “compute” section has also increased.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f291bfe6e68a7b27dffccb07d7d856ad54301564.png" data-download-href="https://discuss.pytorch.org/uploads/default/f291bfe6e68a7b27dffccb07d7d856ad54301564" title="svd_1K_time"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f291bfe6e68a7b27dffccb07d7d856ad54301564_2_690x381.png" alt="svd_1K_time" data-base62-sha1="yBROwIuwJk7FA5OJCxYUVbDLeq8" width="690" height="381" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f291bfe6e68a7b27dffccb07d7d856ad54301564_2_690x381.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f291bfe6e68a7b27dffccb07d7d856ad54301564_2_1035x571.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f291bfe6e68a7b27dffccb07d7d856ad54301564_2_1380x762.png 2x" data-dominant-color="ECEAE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">svd_1K_time</span><span class="informations">1866×1033 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c92301115a5c5f6d495c2a543616e2d44c02b6bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/c92301115a5c5f6d495c2a543616e2d44c02b6bf" title="svd_1M_time"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c92301115a5c5f6d495c2a543616e2d44c02b6bf_2_690x389.png" alt="svd_1M_time" data-base62-sha1="sHkXZEhIjhx0m6DGqfSUSTfKJJ5" width="690" height="389" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c92301115a5c5f6d495c2a543616e2d44c02b6bf_2_690x389.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c92301115a5c5f6d495c2a543616e2d44c02b6bf_2_1035x583.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c92301115a5c5f6d495c2a543616e2d44c02b6bf_2_1380x778.png 2x" data-dominant-color="EBEAE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">svd_1M_time</span><span class="informations">1861×1051 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could anyone help me in understanding why this idle time occurs and why it keeps increasing as we increase the size of the dataset?</p>

username: SimonW
cooked: <p>This is because magma’s implementation of ?gesvd is used under the hood. Unfortunately, the implementation is not purely on GPU and still calls LAPACK/BLAS functions (<a href="https://github.com/maxhutch/magma/blob/master/src/dgesvd.cpp">https://github.com/maxhutch/magma/blob/master/src/dgesvd.cpp</a>). That is probably why you are seeing these idle times on GPU.</p>

username: Sabra_Ossen
cooked: <p>So my understanding here is that PyTorch uses a modified version of magma (which includes more BLAS) and not the original <a href="http://icl.cs.utk.edu/magma/software/index.html" rel="nofollow noopener">magma</a>. Where in the code is the magma function actually invoked, is it <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THC/generic/THCTensorMathMagma.cu#L273" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/THC/generic/THCTensorMathMagma.cu#L273</a>? I would like to get an idea of the path of invocation for SVD.</p>

username: SimonW
cooked: <p>No. magma calls LAPACK/BLAS. PyTorch doesn’t modify it. There are several LAPACK calls in <code>?gesdd</code> of magma <a href="https://github.com/maxhutch/magma/blob/master/src/sgesdd.cpp">https://github.com/maxhutch/magma/blob/master/src/sgesdd.cpp</a>.</p>
<p>Sorry I was linking the wrong file.</p>
<p>And yes, the link you provided is part of the execution path.</p>

username: Sabra_Ossen
cooked: <p>Sorry, I think you might have misunderstood the first line of my reply. What I meant was that as I understood this (<a href="http://icl.cs.utk.edu/magma/software/index.html" rel="nofollow noopener">http://icl.cs.utk.edu/magma/software/index.html</a>) is the magma original library. And this (<a href="https://github.com/maxhutch/magma/blob/master/src/sgesdd.cpp" rel="nofollow noopener">https://github.com/maxhutch/magma/blob/master/src/sgesdd.cpp</a>) is a fork of magma to include more BLAS. The second is what PyTorch uses. Is that correct?</p>
<p>And yes from what I see in the file you pointed out this is not a pure GPU implementation, there are several calls to LAPACK. Thanks very much for the clarification.</p>

username: SimonW
cooked: <p>Oh I see. Sorry about that. I just took the first google result and didn’t look at the owner name <img src="//discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>I believe we use the original magma.</p>

username: Sabra_Ossen
cooked: <p>If so I believe this should be the file (<a href="https://bitbucket.org/icl/magma/src/d646183b0ca94c48ebb8aa1ce75869e98766a605/src/dgesdd.cpp?at=default&amp;fileviewer=file-view-default" rel="nofollow noopener">https://bitbucket.org/icl/magma/src/d646183b0ca94c48ebb8aa1ce75869e98766a605/src/dgesdd.cpp?at=default&amp;fileviewer=file-view-default</a>). I wasn’t able to find a sgesdd.cpp in the magma repo (<a href="https://bitbucket.org/icl/magma/src/d646183b0ca9/src/?at=default" rel="nofollow noopener">https://bitbucket.org/icl/magma/src/d646183b0ca9/src/?at=default</a>).</p>

username: linyu
cooked: <p>Hi, do you solve the problem of GPU is slower than CPU. I encounter the same issue when using the function torch.gesv. Waiting for your apply, thank you!</p>

username: chenhao1umbc
cooked: <p>I got the same issue here…</p>
<p>temp.shape<br>
Out[14]: torch.Size([1140, 1140])<br>
which is low rank structure</p>
<p>u,s,v = torch.svd(temp)<br>
it took 3.5 seconds!<br>
vs<br>
u,s,v = torch.svd(temp.cpu())<br>
it took 0.23 seconds!</p>
<p>ridiculously slow here…</p>

username: CadyWang
cooked: <p>Hello, have you solved this problem?I also encountered, do not know how to solve</p>