Is it possible to run two functions in parrallel under the eager mode during training?

imhuim982: <p>Assume two functions are depending on the tensor x: f1(x) and f2(x)<br>
in f1, it works like:</p>
<pre><code class="lang-auto">void f1(x):
    for layer in layers:
        x=layer(x)
    return x
</code></pre>
<p>Analogous to f2.<br>
I found GPU utility is low to execute f1 and f2 in sequential.<br>
As there is no data dependency for f1 and f2, can they run in parallel to fully leverage GPU?<br>
I know Pytorch is running in async if there is no dataflow dependency. e.g. a1=linear(x); a2=linear(x). They could be executed asynchronizedly by pushing the operations into Cuda flow, right?<br>
But there are data dependencies in the for loops in f1 and f2, could f1(x) and f2(x) be executed asynchronizedly? At least the profiler tell me they are executed in sequential.<br>
I also tried run one of the functions by multi-thread, but the total execution time increased.<br>
So is there any way like torch.jit.fork to run two functions in parallel under eager model during training?</p>

ptrblck: <aside class="quote no-group" data-username="imhuim982" data-post="1" data-topic="191296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/ecccb3/48.png" class="avatar"> imhuim982:</div>
<blockquote>
<p>As there is no data dependency for f1 and f2, can they run in parallel to fully leverage GPU?</p>
</blockquote>
</aside>
<p>Yes, potentially:</p>
<ul>
<li>in different <code>CUDAStream</code>s</li>
<li>if the kernel execution time is longer than the CPU needs to schedule the kernel (could not be the case if the GPU workload is tiny or if your CPU is slow)</li>
<li>if enough compute resources are available on the GPU.</li>
</ul>
<aside class="quote no-group" data-username="imhuim982" data-post="1" data-topic="191296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/ecccb3/48.png" class="avatar"> imhuim982:</div>
<blockquote>
<p>I know Pytorch is running in async if there is no dataflow dependency. e.g. a1=linear(x); a2=linear(x).</p>
</blockquote>
</aside>
<p>PyTorch schedules the kernels and executes them asynchronously <strong>w.r.t. the host</strong>. Both layers will still be executed sequentially by default on the GPU unless you use custom <code>CUDAStream</code>s as described before.</p>
<aside class="quote no-group" data-username="imhuim982" data-post="1" data-topic="191296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/ecccb3/48.png" class="avatar"> imhuim982:</div>
<blockquote>
<p>But there are data dependencies in the for loops in f1 and f2, could f1(x) and f2(x) be executed asynchronizedly?</p>
</blockquote>
</aside>
<p>Since the same, default, <code>CUDAStream</code> is used there are no issues with data dependencies.</p>
<aside class="quote no-group" data-username="imhuim982" data-post="1" data-topic="191296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/ecccb3/48.png" class="avatar"> imhuim982:</div>
<blockquote>
<p>I also tried run one of the functions by multi-thread, but the total execution time increased.</p>
</blockquote>
</aside>
<p>Multithreading in Python is often a bad idea due to the GIL.</p>

imhuim982: <p>Many thanks for your reply! I’ll give a try~</p>

imhuim982: <p>Sorry to bother you again. I tried your suggestion, but I found the total training time increased. I instantiate a CUDA stream instance as follow:</p>
<pre><code class="lang-auto">class A(nn.Module):
    def __init__(self):
        ...
        self.parallel_stream = torch.cuda.Stream()

    def forward(self, x):
        with torch.cuda.stream(self.parallel_stream):
            loss1 = func1(x)
        loss2 = func2(x)
        torch.cuda.synchronize()
        return loss1 + loss2
</code></pre>
<p>Is it possible that the time the CPU needs to schedule the kernel is close to the kernel execution time? The model logic is pretty complicated inside func1 as it works in a dynamic programming way. But there are no operations that need to synchronize with GPU like tensor.to(device).  Is it possible to launch func1 in another stream in a real new thread without GIL (e.g. with the help of C++ of cython) ?</p>
<p>My environment: A100 + Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz</p>

ptrblck: <p>Profile your approach via Nsight Systems and check the kernel execution and overlap on the timeline view.</p>