Cuda Streams in C++ Lib?

username: soulslicer
cooked: <p>I want to do something like this in C++</p>
<pre><code class="lang-auto">s1 = torch.cuda.Stream()
s2 = torch.cuda.Stream()
# Initialise cuda tensors here. E.g.:
A = torch.rand(1000, 1000, device = ‘cuda’)
B = torch.rand(1000, 1000, device = ‘cuda’)
# Wait for the above tensors to initialise.
torch.cuda.synchronize()
with torch.cuda.stream(s1):
    C = torch.mm(A, A)
with torch.cuda.stream(s2):
    D = torch.mm(B, B)
# Wait for C and D to be computed.
torch.cuda.synchronize()
# Do stuff with C and D.
</code></pre>
<p>What is the right way to do this? Say I already have a cudaStream allocated, and I want to do torch::mm, how do I do this?</p>

username: yf225
cooked: <p><code>at::cuda::CUDAStreamGuard guard(stream)</code> is what you are looking for. Example:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/13e6326c07cd44ad52db32ac3918eeb79f8230bc/aten/src/ATen/test/cuda_stream_test.cpp#L152-L162" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/13e6326c07cd44ad52db32ac3918eeb79f8230bc/aten/src/ATen/test/cuda_stream_test.cpp#L152-L162" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/13e6326c07cd44ad52db32ac3918eeb79f8230bc/aten/src/ATen/test/cuda_stream_test.cpp#L152-L162</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="152" style="counter-reset: li-counter 151 ;">
<li>// Setting a stream changes the current device and the stream on that device</li>
<li>{</li>
<li>  at::cuda::CUDAStreamGuard guard(streams1[1]);</li>
<li>  ASSERT_EQ_CUDA(guard.current_device(), at::Device(at::kCUDA, 1));</li>
<li>  ASSERT_EQ_CUDA(at::cuda::current_device(), 1);</li>
<li>  ASSERT_EQ_CUDA(at::cuda::getCurrentCUDAStream(1), streams1[1]);</li>
<li>}</li>
<li>
</li>
<li>// Device and stream are now reset</li>
<li>ASSERT_EQ_CUDA(at::cuda::current_device(), 0);</li>
<li>ASSERT_EQ_CUDA(at::cuda::getCurrentCUDAStream(1), streams1[0]);</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: soulslicer
cooked: <p>so just by declaring at::cuda::CUDAStreamGuard guard(stream), future torch operations will run on that stream? And if I declare my code in {} and it leaves it, it goes back to the default stream?</p>

username: soulslicer
cooked: <p>I tried to run my code with and without the CUDAStreamGuard commented out and the runtime is the same:</p>
<pre><code class="lang-auto">std::vector&lt;at::cuda::CUDAStream&gt; streams;
        streams.emplace_back(at::cuda::getStreamFromPool());
        streams.emplace_back(at::cuda::getStreamFromPool());
        streams.emplace_back(at::cuda::getStreamFromPool());

    measuretime();
    {
        torch::mm_out(X, A, B);
        at::cuda::CUDAStreamGuard g0(streams[0]);
        torch::mm_out(Y, A, C);
        at::cuda::CUDAStreamGuard g1(streams[1]);
        torch::mm_out(Z, A, D);
        at::cuda::CUDAStreamGuard g2(streams[2]);
        torch::mm_out(W, A, E);
    }
    measuretime();
</code></pre>
<p>What am I doing wrong</p>

username: colesbury
cooked: <p>CUDA streams don’t magically speed-up your code. They won’t speed-up most use cases. Mostly they are useful for overlapping copying with compute. They may also be useful if your kernels do not use all the SMs and are not all completely memory bound.</p>
<p>Most likely this is not the case for your code, but it’s hard to guess what the limiting factor is because you don’t specify X, A, B, etc. or measuretime().</p>

username: soulslicer
cooked: <p>Okay, I am yet to profile it with the NVIDIA Profiler…but i eventually will. I assumed there would be a speedup since they dont exactly depend on each other and are reasonably large matrices. Removing each mult operation most definitely cuts 0.1ms off each time, and overall it takes 0.4ms whether I had those streamguard blocks or not.</p>
<p>I just wanted to make sure that I am using the API correctly</p>

username: litttl
cooked: <p>Could you give me some advise?Thanks very much</p><aside class="quote quote-modified" data-post="1" data-topic="99820">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e8c25b/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/help-on-using-stream-in-multithread-to-accelerate/99820">Help on using stream in multithread to accelerate</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    I need some help about using stream in multithread to accelerate my computation. Could you help me and give me some advise? 
my problem: 
the time killed by using multithread and multistreams is seem to using serial loop. it doesn’t speed up. 
my codes: 
I used a thread-pool with 10 threads. I add my task into threads. In thread task funciton, I get a free GPUStream and do some job in this GPUStream. 
the thread task is as follow: 
void* task_thread(void** arg) 
{ 
auto options = torch::TensorOp…
  </blockquote>
</aside>