How can l run two blocks in parallel?

DeepLearner17: <p>Hi,</p>
<p>l would like to run in <strong>parallel (and in the same GPU)</strong> block1(x) and block2(x), which are two independent blocks parameterized by the same input signal x. Here is my forward function.<br>
How can l run <code>x1=self.block1(x)</code> and<code> x2=self.block2(x)</code> in parallel rather than in sequential ?</p>
<p>Thank you</p>
<pre><code class="lang-auto">def  forward(self,x):
       x1=self.block1(x)       
       x2=self.block2(x)      
       x= torch.mm(x1,x2)      
       return x
</code></pre>

albanD: <p>Hi,</p>
<p>The cuda api is asynchronous. So as long as you don’t print the values of the results, they will be pipelined automatically by the cuda driver.<br>
If you have a powerful enough GPU, they might already be running in parallel.</p>

andravin: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, <a class="mention" href="/u/deeplearner17">@DeepLearner17</a></p>
<p>The two modules <code>x1</code> and <code>x2</code> in the example will run sequentially on the same CUDA stream.</p>
<p>Kernels launched on the same stream are not pipelined. All blocks from the first kernel must complete before any blocks from the second kernel can be issued. This is the most basic level of synchronization provided by CUDA.</p>
<p>In order to pipeline two modules, PyTorch would need to run the underlying CUDA kernels on different streams. Then blocks from the second module could begin issuing as soon as all blocks from the first module have issued.</p>
<p>There is a PyTorch API for this. Apparently it respects the stream assignments during the backwards pass, too. See <a href="https://github.com/pytorch/pytorch/pull/8354" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/8354</a></p>
<p>One could imagine a simpler API: Create a module called <code>Parallel</code> that takes a list of of modules and runs them each on a separate stream, and synchronizes at the end. It would looks like this:</p>
<pre><code class="lang-auto">def __init__(self, x):
    ...
    self.branches = nn.Parallel(block1, block2)
    ...

def forward(self, x):
    x1, x2 = self.branches(x, x)
    x = torch.mm(x1, x2)
    return x
</code></pre>
<p>You could build large parallel branches by composing <code>Parallel</code> with <code>Sequential</code>.</p>
<p>I think this API might be simpler than using streams directly, because we usually don’t care which streams are actually used for different modules, we just want to indicate the parallelism in the model and have the framework exploit it automatically.</p>
<p>Anyway hope this helps somebody.</p>

albanD: <p>In practice, most of our kernels actually use the whole GPU and so kernels don’t run at the same time.<br>
So we see benefits from using stream only in very niche case, and general users wouldn’t need it.<br>
We only provide the API for advanced user that know that their workload is in this niche case and they want to slightly improve the performances.</p>

andravin: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<aside class="quote no-group" data-username="albanD" data-post="4" data-topic="61618">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>In practice, most of our kernels actually use the whole GPU and so kernels don’t run at the same time.</p>
</blockquote>
</aside>
<p>That’s not exactly how GPUs work. After a kernel issues all of its blocks, there is a “tail” of blocks that partially occupy the GPU. If there are just a few waves of blocks, then the tail can be a significant fraction of the total kernel execution time. Another kernel launched on a second stream could fill the device during the tail of the first kernel, increasing device utilization.</p>
<p>The issue for PyTorch would be that kernels that have a very short execution time will hit the limit of how fast you can launch kernels from the CPU (let alone from a python application).</p>
<p>CUDA Graphs were introduced to move kernel launch to the device side, reducing launch overhead for short running kernels.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/blog/cuda-graphs/">
  <header class="source">
      <img src="https://developer-blogs.nvidia.com/wp-content/themes/nvidia/dist/images/favicon_300a1064.ico" class="site-icon" width="48" height="48">

      <a href="https://developer.nvidia.com/blog/cuda-graphs/" target="_blank" rel="noopener nofollow ugc" title="11:39PM - 05 September 2019">NVIDIA Technical Blog – 5 Sep 19</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:299/405;"><img src="https://developer-blogs.nvidia.com/wp-content/uploads/2019/09/CUDA-Graphs.png" class="thumbnail" width="299" height="405"></div>

<h3><a href="https://developer.nvidia.com/blog/cuda-graphs/" target="_blank" rel="noopener nofollow ugc">Getting Started with CUDA Graphs | NVIDIA Technical Blog</a></h3>

  <p>The performance of GPU architectures continue to increase with every new generation. Modern GPUs are so fast that, in many cases of interest, the time taken by each GPU operation (e.g.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Apparently there is an effort underway to use CUDA Graphs in PyTorch,</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/15623">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/15623" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/15623" target="_blank" rel="noopener nofollow ugc">Static Graphs using CUDA 10 Graphs API</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-12-30" data-time="03:15:06" data-timezone="UTC">03:15AM - 30 Dec 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2021-09-11" data-time="00:51:26" data-timezone="UTC">12:51AM - 11 Sep 21 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/fps7806" target="_blank" rel="noopener nofollow ugc">
          <img alt="fps7806" src="https://avatars.githubusercontent.com/u/7218057?v=4" class="onebox-avatar-inline" width="20" height="20">
          fps7806
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          feature
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cuda
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cuda graphs
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🚀 Feature
CUDA 10 released a new feature called CUDA Graphs which allows you<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden"> to build static graphs that can minimizes the overhead of launching multiple kernels. The API comes with functions that allow you to capture a stream (multiple streams are also supported) and transform it into a CUDA graph. Exposing this feature to pytorch can be very beneficial to many applications.

## Motivation
When working with small kernels it is important to minimize any overhead without adding complexity.

## Pitch

I propose something like:
```
g = CUDAGraph()
with g.capture():
    # Static computation that can be combined into 1 graph
    ....

while True:
    g.execute()
```

## Alternatives

Currently Pytorch offers JIT, and C++ APIs that mitigate the overhead. The CUDA Graph approach is theoretically faster and IMHO more friendly to users. 

## Additional context

I have a small demo working where I run a small neural network 1000 times and here are the current benchmarks I have:
```
Pytorch c++ Front-end: 152.633ms
Pytorch c++ with cuda graphs capture:136.692ms
Pytorch Python: 354.79ms
Pytorch Python + cuda graphs capture: 134.29ms
```


cc @ngimel</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>… but it seems to have hit a snag:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/21589">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/21589" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/21589" target="_blank" rel="noopener nofollow ugc">Some parts of PyTorch test suite don't work properly on non-default stream</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-06-10" data-time="14:43:34" data-timezone="UTC">02:43PM - 10 Jun 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-05-11" data-time="17:50:42" data-timezone="UTC">05:50PM - 11 May 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/ezyang" target="_blank" rel="noopener nofollow ugc">
          <img alt="ezyang" src="https://avatars.githubusercontent.com/u/13564?v=4" class="onebox-avatar-inline" width="20" height="20">
          ezyang
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cuda
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: tests
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          large
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">In @fbhuba PR https://github.com/pytorch/pytorch/pull/21474 we switch the stream<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden"> before running tests. But in this PR, a lot of tests failed and we had to disable the stream switching behavior for large portions of the test suite. There is probably something very important in PyTorch that is not stream safe, we need to figure it out.

Currently known reasons why you might be on wrong stream:

* Calling thrust functions (e.g., `thrust::tabulate`) without specifying a policy that puts things on the right stream
* Backwards runs on default stream</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Anyway, as these issues are very interesting to me, I would like to learn more about what the roadmap looks like for this kind of functionality in PyTorch.</p>
<p>[Edit: fixed link to <span class="hashtag-raw">#15623</span>]</p>

albanD: <p>cc <a class="mention" href="/u/ngimel">@ngimel</a> that will know better what are the plans for such features.</p>

ngimel: <p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a> as cuda graphs effort is driven by nvidia, so he knows more about roadmaps. <span class="hashtag">#21589</span> did not hit a snag, in fact, we believe that now pytorch is stream-safe and non-default streams can be used as needed. Some operations incure host-device synchronization and thus would break graph capture, but the majority should be good. Biggest problem for enabling cuda graphs last time I was involved in this was caching allocator - cuda graph wants kernel arguments (including data pointers) to remain the same between invocations, and with caching allocator we cannot guarantee that.<br>
As for tail effect, indeed, streams would help there, but it is still a pretty niche usecase (you need a kernel that has just a few waves for the tail to be a noticeable part of execution time), which translates into a fairly narrow range of sizes where it would be observable.</p>

andravin: <p>The “snag” was meant to refer to “Static Graphs using CUDA 10 Graphs API <span class="hashtag">#15623</span>”, I updated my comment with the correct link.</p>
<p>I’m not sure that low achieved occupancy is so uncommon. A small model (eg MobileNetV2) on a big GPU will have just a few waves per kernel even for a moderate batch size. Small batch size is especially interesting for low latency inference. It is also possible that choosing a small batch size could improve cache utilization.</p>
<p>I guess I need to learn more about the PyTorch plan for efficient inference on state of the art vision models, efficient training of small models on big GPUs, etc.</p>