Unable to Execute Forward and Backward Functions in Parallel Across Different CUDA Streams(pytorch1.9)

lssyes: <p>Hello,</p>
<p>I’m experiencing issues trying to execute my forward and backward propagation functions in parallel using PyTorch and CUDA streams. My code creates two different streams and attempts to execute the forward and backward propagation operations separately within these streams, but they do not seem to run in parallel.</p>
<p>Here is the relevant part of my code:</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch.cuda import nvtx
from contextlib import nullcontext


MB = 1 &lt;&lt; 20
DTYPE = torch.FloatTensor
BS = 10000
REPEAT = 10
MBS = 1000

BUF_SHAPE = (256, 256)

print(f"BUF_SHAPE: {BUF_SHAPE}")

mm_parameter = torch.randn(BUF_SHAPE[0], BUF_SHAPE[0]).cuda()

def forward(input):
     return input * mm_parameter
def backward(input):
    return input * mm_parameter
 
def main():
    test_stream = True
    if test_stream:
        stream1 = torch.cuda.Stream(priority=-1)
        stream2 = torch.cuda.Stream(priority=0)
    else:
        stream1 = torch.cuda.default_stream()
        stream2 = torch.cuda.default_stream()

    PROFILE = True
    


    data = torch.ones((BS, *BUF_SHAPE)).cuda()
    print(f"Data size: {data.element_size() * data.nelement() / MB} MB")
    
    input_list = torch.chunk(data, REPEAT, dim=0)    
    output_list = []

    
    torch.cuda.synchronize()

    forward_start_event = [torch.cuda.Event(enable_timing=True, blocking=True) for _ in range(REPEAT)]
    forward_complete_event = [torch.cuda.Event(enable_timing=True, blocking=True) for _ in range(REPEAT)]
    backward_start_event = [torch.cuda.Event(enable_timing=True, blocking=True) for _ in range(REPEAT)]
    backward_complete_event = [torch.cuda.Event(enable_timing=True, blocking=True) for _ in range(REPEAT)]
    
    def run(x_data):
        output_list = []
        
        for iteration in range(REPEAT):
            with nvtx.range(f"iteration-{iteration}"):
                with torch.cuda.stream(stream1):
                    with nvtx.range(f"forward-{iteration}"):
                        f_out = forward(x_data[iteration])
                        nvtx.mark(f"F-{iteration} done")
                    
                            
                # print(f"output: {output.cpu().numpy()[0]}")
                with torch.cuda.stream(stream2):
                    with nvtx.range(f"backward-{iteration}"):
                        output_list.append(backward(f_out))
                        nvtx.mark(f"B-{iteration} done")
                    
                
        torch.cuda.synchronize()
        return output_list
                    
                    
                    
    import time
                        
    run(input_list)
    run(input_list)
    torch.cuda.synchronize()
    start = time.time()
    for epoch in range(10):
        print(f"epoch: {epoch}")
        with nvtx.range(f"Epoch_{epoch}"):
            input_list = run(input_list)

        
    torch.cuda.synchronize()
    print("Finished")
    print(f"time: {(time.time() - start)/10}")
   
    
    
    
               
    torch.cuda.synchronize()
    
if __name__ == "__main__":
    with torch.cuda.profiler.profile():
        main()
</code></pre>
<p>Here is the relevant result of my nsys:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd435ac0f4d467e2fe62e955eba2019db5b7ba42.png" data-download-href="https://discuss.pytorch.org/uploads/default/dd435ac0f4d467e2fe62e955eba2019db5b7ba42" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd435ac0f4d467e2fe62e955eba2019db5b7ba42_2_690x239.png" alt="image" data-base62-sha1="vznPigYjTyM2s3CqwAuVwA3fa9Q" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd435ac0f4d467e2fe62e955eba2019db5b7ba42_2_690x239.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd435ac0f4d467e2fe62e955eba2019db5b7ba42_2_1035x358.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd435ac0f4d467e2fe62e955eba2019db5b7ba42_2_1380x478.png 2x" data-dominant-color="CBD4CF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1380×478 63.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>At the same time, I adjusted the priorities of the two streams to be the same, but I still couldn’t achieve pipelined parallelism between the forward and backward operations.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8cb16530a4d69f8f78c3dc9e0383d08d101aa69d.png" data-download-href="https://discuss.pytorch.org/uploads/default/8cb16530a4d69f8f78c3dc9e0383d08d101aa69d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8cb16530a4d69f8f78c3dc9e0383d08d101aa69d_2_690x240.png" alt="image" data-base62-sha1="k4CUDqGR3KLViK6ArKwsu0lpQQJ" width="690" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8cb16530a4d69f8f78c3dc9e0383d08d101aa69d_2_690x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8cb16530a4d69f8f78c3dc9e0383d08d101aa69d_2_1035x360.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8cb16530a4d69f8f78c3dc9e0383d08d101aa69d_2_1380x480.png 2x" data-dominant-color="C6D1CE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1380×480 76.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have verified that both functions do not run simultaneously before calling torch.cuda.synchronize(). I expected these operations to run in parallel in different streams, but they appear to be executed sequentially. What could be the reason for this? Are there any suggested solutions?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/running-independent-nn-module-instances-in-nn-modulelist-truly-in-parallel-in-pytorch/188166/3">This post</a> might be helpful.</p>

lssyes: <p>Thank you very much for your response!</p>
<p>I appreciate the reference and will certainly look into the post you mentioned for potential solutions. Meanwhile, could you guide me on how to use Nsight Systems to determine if the lack of sufficient SM resource is causing the inability to parallelize the backward pass of the current iteration with the forward pass of the next iteration? I’m trying to understand if hardware limitations might be influencing the observed behavior or if there are adjustments I could make in the way I structure or synchronize my streams.</p>
<p>Thank you once again for your help!</p>