cudaLaunchKernel takes 99% of CPU time

AimlessArrow: <p>I try to profile the text detection model from <a href="https://github.com/ymy-k/dptext-detr" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ymy-k/DPText-DETR: DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer (AAAI 2023)</a> using torch.profiler<br>
According to the results cudaLaunchKernel takes 99.19% of CPU time.<br>
I tried to use other non-Pytorch-related profilers, that shows that <a href="https://github.com/ymy-k/DPText-DETR/blob/df5ddb526cbc81059a0610980cae151ac840950b/adet/layers/deformable_transformer.py#L115" rel="noopener nofollow ugc">this part</a> is the one to consume the most part of the time.</p>
<pre><code class="lang-auto">    def gen_encoder_output_proposals(self, memory, memory_padding_mask, spatial_shapes):
        N_, S_, C_ = memory.shape
        base_scale = 4.0
        proposals = []
        _cur = 0
        for lvl, (H_, W_) in enumerate(spatial_shapes):
            mask_flatten_ = memory_padding_mask[:, _cur:(_cur + H_ * W_)].view(N_, H_, W_, 1)
            valid_H = torch.sum(~mask_flatten_[:, :, 0, 0], 1)
            valid_W = torch.sum(~mask_flatten_[:, 0, :, 0], 1)

            grid_y, grid_x = torch.meshgrid(torch.linspace(0, H_ - 1, H_, dtype=torch.float32, device=memory.device),
                                            torch.linspace(0, W_ - 1, W_, dtype=torch.float32, device=memory.device))
            grid = torch.cat([grid_x.unsqueeze(-1), grid_y.unsqueeze(-1)], -1)

            scale = torch.cat([valid_W.unsqueeze(-1), valid_H.unsqueeze(-1)], 1).view(N_, 1, 1, 2)
            grid = (grid.unsqueeze(0).expand(N_, -1, -1, -1) + 0.5) / scale
            wh = torch.ones_like(grid) * 0.05 * (2.0 ** lvl)
            proposal = torch.cat((grid, wh), -1).view(N_, -1, 4)
            proposals.append(proposal)
            _cur += (H_ * W_)
        output_proposals = torch.cat(proposals, 1)
        output_proposals_valid = ((output_proposals &gt; 0.01) &amp; (output_proposals &lt; 0.99)).all(-1, keepdim=True)
        output_proposals = torch.log(output_proposals / (1 - output_proposals))
        output_proposals = output_proposals.masked_fill(memory_padding_mask.unsqueeze(-1), float('inf'))
        output_proposals = output_proposals.masked_fill(~output_proposals_valid, float('inf'))

        output_memory = memory
        output_memory = output_memory.masked_fill(memory_padding_mask.unsqueeze(-1), float(0))
        output_memory = output_memory.masked_fill(~output_proposals_valid, float(0))
        output_memory = self.enc_output_norm(self.enc_output(output_memory))
        return output_memory, 
</code></pre>
<p>So, I tried to play around this part, creating some random tensors like this:</p>
<pre><code class="lang-auto">memory = torch.randn([1, shape, 256], device = 'cuda')
memory_padding_mask = torch.randn([1, shape], device = 'cuda') &gt; 0.5
spatial_shapes = torch.randint(16, 224, [4, 2], device = 'cuda')
spatial_shapes = torch.tensor([[125, 125],
        [ 63,  63],
        [ 32,  32],
        [ 16,  16]], device='cuda')
tensors.append((memory, memory_padding_mask, spatial_shapes))

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], with_stack=True,) as prof:
    with record_function("model_inference"):
        for memory, memory_padding_mask, spatial_shapes in tensors:
            r = gen_encoder_output_proposals(memory, memory_padding_mask, spatial_shapes)
</code></pre>
<p>And I got the same result. Warming up the model doesn’t help me to improve the performance.</p>
<p>However, if I firstly run it on e.g. tensors[:1] and then I execute the whole loop, I got way better results.</p>
<p>So whether there is a weak part that should be replaced or is it a problem of incorrect profiling?</p>

ptrblck: <p>For a more detailed understanding of your workload I would recommend to profile your code with e.g. Nsight Systems as described <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">here</a>. In the timeline view you would be able to see if your workload is CPU-limited and where the bottlenecks are.</p>

AimlessArrow: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the recomendation.<br>
I tried to use nsight and got the following result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d6883fd0caea61b5c00361d21e00f4e0997db78.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9d6883fd0caea61b5c00361d21e00f4e0997db78" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d6883fd0caea61b5c00361d21e00f4e0997db78_2_690x252.jpeg" alt="image" data-base62-sha1="msuSXxJwKmxned7IdF9RvmvdFnW" width="690" height="252" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d6883fd0caea61b5c00361d21e00f4e0997db78_2_690x252.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d6883fd0caea61b5c00361d21e00f4e0997db78_2_1035x378.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d6883fd0caea61b5c00361d21e00f4e0997db78_2_1380x504.jpeg 2x" data-dominant-color="E2E6E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1588×580 205 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So, I still wonder if it is okay for cudaLaunchKernel to take the most part of Total Time?</p>

ptrblck: <p>What do you see in the timeline? Are all kernels packed or are you seeing white spaces between their launches? Were you able to isolate the bottleneck?</p>

qixiang: <p>Hi, recently I have met almost the same problem. I’ve tried to profile two diffferent models and result show that <strong>all the first cudaLaunchKernel</strong> cost most of time. I am using Torch 1.10 and torch vision 0.10</p>

ptrblck: <p>The very first calls are irrelevant as they might represent the warmup phase. Especially if your GPU was at its idle state the first op will cost more.</p>

qixiang: <p>Thanks for your relying! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I did run a few times (about 10 to 15) forward for warmup. However, since I am not clear with the definition of it, I am not sure whether this is enough for warmup.</p>
<p>I warmup like this:</p>
<p>warmup_time=15<br>
for data, i in enumerate(dataloader):<br>
model(data)<br>
if i == warmup_time:<br>
# profile<br>
break<br>
The tracing is like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/653f8b63cf320ec8b937338ad1e1825cc461ffac.png" data-download-href="https://discuss.pytorch.org/uploads/default/653f8b63cf320ec8b937338ad1e1825cc461ffac" title="IMG_7585"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653f8b63cf320ec8b937338ad1e1825cc461ffac_2_690x248.png" alt="IMG_7585" data-base62-sha1="erGmOrp2lF6FmF5LjVoU5O8Hayw" width="690" height="248" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653f8b63cf320ec8b937338ad1e1825cc461ffac_2_690x248.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653f8b63cf320ec8b937338ad1e1825cc461ffac_2_1035x372.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653f8b63cf320ec8b937338ad1e1825cc461ffac_2_1380x496.png 2x" data-dominant-color="EBDDE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_7585</span><span class="informations">1454×523 36.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thank you!</p>