Does the PyTorch profiler serialize computations on cuda?

username: dmadeka1
cooked: <p>I’ve been trying to profile some of my code, and have noticed that it seemed awfully serial. So after a lot of deep dive, I came down to a single example I thought was relevant</p>
<pre><code class="lang-auto">import torch
import time

from torch.profiler import profile, record_function, ProfilerActivity

if __name__ == "__main__":

    N = 1000
    mul = 50
    num2 = 50

    profile_it = True

    streams = [torch.cuda.Stream() for _ in range(8)]

    for i in range(8):
        with torch.cuda.stream(streams[i]):
            torch.rand(10, 10, device='cuda:{}'.format(i))

    print(N)
    q2 = []
    qs = []
    for i in range(8):
        with torch.cuda.stream(streams[i]):
            temp = torch.rand(mul * N, mul * N, device='cuda:{}'.format(i))
            qs.append(temp)

    for i in range(8):
        temp = 'cuda:{}'.format(i)
        torch.cuda.synchronize(torch.device(temp))
        print(temp)


    if profile_it == True: 
        with profile(activities=[ProfilerActivity.CUDA]) as prof2:
            for i, q in enumerate(qs):
                with torch.cuda.stream(streams[i]):
                        q2.append(q @ q @ q @ q)

    else:
        for i, q in enumerate(qs):
            with torch.cuda.stream(streams[i]):
                q2.append(q @ q @ q @ q)


    for i in range(8):
        temp = 'cuda:{}'.format(i)
        torch.cuda.synchronize(torch.device(temp))
        print(temp)

    if profile_it:
        prof2.export_chrome_trace('check.json')

</code></pre>
<p>So when I keep the profiler off, this is what my gpu usage looks like:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/0533999a8636f05a036a29ae41af3d2ec61cc30e.png" data-download-href="https://discuss.pytorch.org/uploads/default/0533999a8636f05a036a29ae41af3d2ec61cc30e" title="torchnoprofile"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0533999a8636f05a036a29ae41af3d2ec61cc30e_2_690x251.png" alt="torchnoprofile" data-base62-sha1="K0WgRXjKdj41kxE8xZWolG59kW" width="690" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0533999a8636f05a036a29ae41af3d2ec61cc30e_2_690x251.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0533999a8636f05a036a29ae41af3d2ec61cc30e_2_1035x376.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0533999a8636f05a036a29ae41af3d2ec61cc30e_2_1380x502.png 2x" data-dominant-color="3D152D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torchnoprofile</span><span class="informations">1844×672 152 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I turn it on, here’s what it looks like:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99d4ef962ec7e6e7d5da7393acfb45312f6c042e.png" data-download-href="https://discuss.pytorch.org/uploads/default/99d4ef962ec7e6e7d5da7393acfb45312f6c042e" title="torchwithprofile"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99d4ef962ec7e6e7d5da7393acfb45312f6c042e_2_690x150.png" alt="torchwithprofile" data-base62-sha1="lWRguxp6fRwzcomFn6BNpewkxpY" width="690" height="150" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99d4ef962ec7e6e7d5da7393acfb45312f6c042e_2_690x150.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99d4ef962ec7e6e7d5da7393acfb45312f6c042e_2_1035x225.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/99d4ef962ec7e6e7d5da7393acfb45312f6c042e_2_1380x300.png 2x" data-dominant-color="411C30"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torchwithprofile</span><span class="informations">1794×392 146 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For the Nsight profiler using <a href="https://gist.github.com/mcarilli/376821aa1a7182dfcf59928a7cde3223" rel="noopener nofollow ugc">this</a>, it looks like:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90a50128ee31e8699cc3635f2c5eb628adf86dca.png" data-download-href="https://discuss.pytorch.org/uploads/default/90a50128ee31e8699cc3635f2c5eb628adf86dca" title="nsight"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90a50128ee31e8699cc3635f2c5eb628adf86dca_2_690x174.png" alt="nsight" data-base62-sha1="kDAh8b93acyQjaPez6LnjOCsP22" width="690" height="174" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90a50128ee31e8699cc3635f2c5eb628adf86dca_2_690x174.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90a50128ee31e8699cc3635f2c5eb628adf86dca_2_1035x261.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/90a50128ee31e8699cc3635f2c5eb628adf86dca_2_1380x348.png 2x" data-dominant-color="3F192F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nsight</span><span class="informations">1948×494 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With a profile trace that looks right:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c33fd070f015c61e6992e78b7366a6af6e187ade.png" data-download-href="https://discuss.pytorch.org/uploads/default/c33fd070f015c61e6992e78b7366a6af6e187ade" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c33fd070f015c61e6992e78b7366a6af6e187ade_2_690x235.png" alt="image" data-base62-sha1="rRfPl8KxYIyubTR79UdX3G9L3Ii" width="690" height="235" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c33fd070f015c61e6992e78b7366a6af6e187ade_2_690x235.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c33fd070f015c61e6992e78b7366a6af6e187ade_2_1035x352.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c33fd070f015c61e6992e78b7366a6af6e187ade_2_1380x470.png 2x" data-dominant-color="363A38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2944×1004 235 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this expected behavior? I couldnt figure out from the doc strings if I should expect this -</p>

username: ptrblck
cooked: <p>I wouldn’t expect to see a serialization of workloads using the native PyTorch profiler, but I’m also using Nsight Systems for my profiling.<br>
Could you export the <code>trace</code> from the native profiler and also visualize it as it should also show the overlaps (or not in case something weird is indeed going on)?</p>

username: dmadeka1
cooked: <p>Sorry <a class="mention" href="/u/ptrblck">@ptrblck</a> , I exported but forgot to attach - attaching a screenshot, which corresponds to what gpustat -i shows</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0fefeda436e6ec6e4f85f6e9985a3b3dac5afe1.png" data-download-href="https://discuss.pytorch.org/uploads/default/d0fefeda436e6ec6e4f85f6e9985a3b3dac5afe1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0fefeda436e6ec6e4f85f6e9985a3b3dac5afe1_2_690x224.png" alt="image" data-base62-sha1="tORDDcDlRvUANUhAnMrVWUeHR6h" width="690" height="224" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0fefeda436e6ec6e4f85f6e9985a3b3dac5afe1_2_690x224.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0fefeda436e6ec6e4f85f6e9985a3b3dac5afe1_2_1035x336.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0fefeda436e6ec6e4f85f6e9985a3b3dac5afe1_2_1380x448.png 2x" data-dominant-color="E4E5E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3278×1068 244 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: dmadeka1
cooked: <p>This might be related to <a href="https://discuss.pytorch.org/t/how-does-async-copy-with-without-streams-work/176120">176120</a> as well</p>

username: ptrblck
cooked: <p>Thanks for sharing the outputs. I’ll try to reproduce the outputs later on one of our systems, as I don’t know how to properly interpret the native PyTorch profiler output since I’m rarely using its visualization.</p>