Doing QR decomposition on GPU is much slower than on CPU

username: YiifeiWang
cooked: <p>Pytorch has its function <em>torch.qr()</em> to implement QR decomposition. For instance, let A be a matrix with size of 3000*60. I have installed Magma as LAPACK support. If I place A on CPU by <em>A.cpu()</em>, to compute <em>torch.qr(A)</em> for 100 times, it will take about 0.4 second. However, when I place A on GPU by <em>A.cuda()</em>, it will take 1.4 seconds. I thought running on GPU should be faster than running on CPU. Can anyone explain this phenomenon to me? Thanks a lot!</p>

username: Tank
cooked: <p>One possible reason is that it takes time to send the data to the GPU. When a calculation is very quick as in this case there are some other small time costs that could add up. If you want to check whether what I am saying is correct try and do a QR decomposition on a matrix that takes a few minutes to run on a cpu and then try it on the GPU.</p>

username: YiifeiWang
cooked: <p>Thank you for your reply! Actually I want to compute the QR decompositon of small matrices many times instead of a big matrix one time. So I guess what you mean is that torch.qr() on GPU is more suitable to do QR decomposition of a big matrix?</p>

username: Tank
cooked: <p>Not necessarily. I am not an expert in this area. However, I believe it depends on how you are passing the small matrices to the GPU. There is probably an upfront cost of sending something to a GPU, if you manage to send all the small matrices at once it might be quicker in the GPU. If you have something like a for loop where at each iteration you send the individual matrix it will probably be slower in GPU than CPU.</p>

username: YiifeiWang
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/618771f132e20fe7148f0fdfcebcddb28041fec3.png" data-download-href="https://discuss.pytorch.org/uploads/default/618771f132e20fe7148f0fdfcebcddb28041fec3" title="test"><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/618771f132e20fe7148f0fdfcebcddb28041fec3.png" alt="test" data-base62-sha1="dUMvgT3RTaDAUFdvRUrFibFzfCX" width="657" height="499" data-dominant-color="DDDDDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test</span><span class="informations">742×564 67.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I test the time to transfer data between CPU and GPU. And from the result, it seems that the time cost of transferring data can be neglected compared to the time to do the QR decomposition. It is weird that GPU performs much worse on QR decomposition of small matrices compared to CPU. As in the picture, the time to do QR over a 100<em>600 matrix is close to the time to do QR over a 100</em>6000 matrix.</p>

username: ptrblck
cooked: <p>How did you measure the transfer time?<br>
Note that CUDA calls are asynchronous, so that the data transfer and processing will be performed in the background.<br>
If you want to time the transfer time, you should synchronize the calls with: <code>torch.cuda.synchronize()</code>:</p>
<pre><code class="lang-python">a = torch.randn(...)
torch.cuda.synchronize()
t0 = time.time()
a = a.to('cuda:0')
torch.cuda.synchronize()
t1 = time.time()
</code></pre>

username: YiifeiWang
cooked: <p>Based on your suggestion of measuring transfer time, my current code is</p>
<pre><code class="lang-auto">    ns=[600,6000,6000,60000,60000]
    for i, p in enumerate(ps):
        n=ns[i]
        A=torch.rand(p,n).cpu()
        torch.cuda.synchronize()
        t0 = time.time()
        A.qr()
        torch.cuda.synchronize()
        t1 = time.time()
        t=t1-t0
        print('%d*%d cpu: time_cost %f'%(p,n,t))
        A=torch.rand(p,n).cuda()
        torch.cuda.synchronize()
        t0 = time.time()
        A.qr()
        torch.cuda.synchronize()
        t1 = time.time()
        t=t1-t0
        print('%d*%d cuda: time_cost %f'%(p,n,t))
        A = torch.randn(p,n).cpu()
        torch.cuda.synchronize()
        t0 = time.time()
        A = A.to('cuda:0')
        torch.cuda.synchronize()
        t1 = time.time()
        t=t1-t0
        print('%d*%d cpu-to-cuda: time_cost %f'%(p,n,t))
        A=torch.rand(p,n).cuda()
        torch.cuda.synchronize()
        t0 = time.time()
        A=A.cpu()
        torch.cuda.synchronize()
        t1 = time.time()
        t=t1-t0
        print('%d*%d cuda-to-cpu: time_cost %f'%(p,n,t))
</code></pre>
<p>And the result is the following picture:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/ab834e805543295ce84d509ed79181de86cb15fa.png" data-download-href="https://discuss.pytorch.org/uploads/default/ab834e805543295ce84d509ed79181de86cb15fa" title="23"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/ab834e805543295ce84d509ed79181de86cb15fa.png" alt="23" data-base62-sha1="otgYb89M4yPFisvnaNhho7xDpBE" width="558" height="499" data-dominant-color="DADADA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">23</span><span class="informations">630×564 65.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: eusouoandre
cooked: <p>Weird that cpu / cuda time_cost  for 100*6000 matrix. Cuda is at least 2x slower than gpu.</p>
<p><strong>Confirmed bug !</strong> <a href="https://github.com/pytorch/pytorch/issues/22573#issuecomment-509657100" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/22573#issuecomment-509657100</a></p>