Kernels launched to different cuda streams serialized

username: Omry_Yadan
cooked: <p>Sample code:<br>
<a href="http://paste.ubuntu.com/p/Bw9mgtb9sD/" class="onebox" target="_blank" rel="noopener nofollow ugc">http://paste.ubuntu.com/p/Bw9mgtb9sD/</a></p>
<p>Running nvprof with the following command:<br>
<code>/usr/local/cuda/bin/nvprof --concurrent-kernels on --print-api-summary --print-gpu-summary --output-profile profile.nvvp -f --profile-from-start off --track-memory-allocations on --demangling  on  --trace gpu,api python stream.py</code></p>
<p>Opening nvvp, I see that the kernels runing on the 5 streams one after the other, instead of all at the same time.<br>
This seems wrong, am I misusing the API or is there some other problem?</p>
<p>System info:</p>
<pre><code class="lang-auto">
PyTorch version: 1.0.1.post2
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Quadro P5000
GPU 1: Quadro P5000

Nvidia driver version: 418.39
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.15.4
[pip] torch==1.0.1.post2
[pip] torchfile==0.1.0
[pip] torchvision==0.2.2
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.1                      144  
[conda] mkl_fft                   1.0.6            py36hd81dba3_0  
[conda] mkl_random                1.0.2            py36hd81dba3_0  
[conda] pytorch                   1.0.1           py3.6_cuda10.0.130_cudnn7.4.2_2    pytorch
[conda] torchfile                 0.1.0                     &lt;pip&gt;
[conda] torchvision               0.2.2                      py_3    pytorch
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f827554702b232044f8ad8a990e77096e21551d6.png" data-download-href="https://discuss.pytorch.org/uploads/default/f827554702b232044f8ad8a990e77096e21551d6" title="Screenshot%20at%202019-03-15%2011%3A28%3A54"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f827554702b232044f8ad8a990e77096e21551d6_2_690x284.png" alt="Screenshot%20at%202019-03-15%2011%3A28%3A54" data-base62-sha1="zpgGniGX68VoUxMBYamSrwLG6Xk" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f827554702b232044f8ad8a990e77096e21551d6_2_690x284.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f827554702b232044f8ad8a990e77096e21551d6_2_1035x426.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f827554702b232044f8ad8a990e77096e21551d6_2_1380x568.png 2x" data-dominant-color="EEEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20at%202019-03-15%2011%3A28%3A54</span><span class="informations">1699×701 56.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: smth
cooked: <p>It’s likely that the cublas launches are occupying the whole GPU, so there’s no room for stream overlap.</p>

username: Omry_Yadan
cooked: <p>In the example I used a matrix that was bigger than what I actually motivated me to look into the cuda streams approach.<br>
what actually happens in my use case (w = 500) is that kernel launches dominates the time because they are running in a single thread using a single CPU core.</p>

username: cgn
cooked: <p>Hi, I just found the same problems. Do you have any solution now? Besides, when I use torch.autograd.profiler.emit_nvtx(), nvvp only shows the default streams, no other streams.</p>

username: Omry_Yadan
cooked: <p>If I remember correctly, the conclusion was that this is caused by launching too small kernels.<br>
The CUDA launch overhead for such small kernels is too high.</p>