Nms gpu cost time so more than caffe

username: Ryan95
cooked: <p>Hi, have anyone compared the GPU time spent on NMS in caffe and pytorch? I recently transfer a model from caffe to libtorch. And I found that the main time spent by libtorch is concentrated on the NMS operation. the time is 160ms, which takes 80% of all forward time. but the overall time consumption of caffe is only 150ms!<br>
The nms implementation I used is copying from torchvision0.3, and the maskrcnn-benchmark is also around 160ms. so Is there an operation that can continue to optimize nms? thank you</p>

username: tom
cooked: <p>How did you post this twice?</p>
<p>I thought that the PyTorch NMS was ultimately derived from the (caffe2) detectron, so it seems strange that it should be that much slower.<br>
You would have to show your benchmarking code before I would try to find out why it is different.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Ryan95
cooked: <p>Hi,Tommas. I also want to post it, but unfortunately I am doing pvanet’s libtorch implementation, nms is only a small part of it. it may not be possible to upload the entire project. I just used the c++ chrono library to record the time spent in the nms function and compare it with the caffe version of pvanet. sorry for can’t provide you with more information. <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>

username: tom
cooked: <p>Yeah, well, the usual way to do these things is to grab an input and try to measure the function in isolation.<br>
Benchmarking has quite a few pitfalls, in particular with CUDA asynchronous computation involved, so it’s hard to say whether you found something where PyTorch is indeed terribly slow or whether you’ve just screwed up your benchmarking (I have certainly done that before, too).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Ryan95
cooked: <p>ok,I understand, Thanks a lot!</p>

username: Ryan95
cooked: <p>Hi,tom. this is nms code that i copy from torchvision 0.3.<br>
I found nms_kernel function is very fast, it’s only cost 0.01ms, but this move data operation</p>
<blockquote>
<p>at::Tensor mask_cpu = mask.to(at::kCPU);</p>
</blockquote>
<p>from gpu to cpu is very slow, because it spent 222.432ms. can i do something to reduce time consumption?</p>
<pre><code class="lang-auto">// boxes is a N x 5 tensor
at::Tensor nms_cuda(const at::Tensor boxes, float nms_overlap_thresh) {
  using scalar_t = float;
  AT_ASSERTM(boxes.type().is_cuda(), "boxes must be a CUDA tensor");
  at::cuda::CUDAGuard device_guard(boxes.device());

  auto scores = boxes.select(1, 4);
  auto order_t = std::get&lt;1&gt;(scores.sort(0, /* descending=*/true));
  auto boxes_sorted = boxes.index_select(0, order_t);

  int boxes_num = boxes.size(0);

  const int col_blocks = at::cuda::ATenCeilDiv(boxes_num, threadsPerBlock);

  at::Tensor mask =
      at::empty({boxes_num * col_blocks}, boxes.options().dtype(at::kLong));

  dim3 blocks(col_blocks, col_blocks);
  dim3 threads(threadsPerBlock);
  cudaStream_t stream = at::cuda::getCurrentCUDAStream();

  AT_DISPATCH_FLOATING_TYPES_AND_HALF(
      boxes_sorted.type(), "nms_kernel_cuda", [&amp;] {
        nms_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads, 0, stream&gt;&gt;&gt;(
            boxes_num,
            nms_overlap_thresh,
            boxes_sorted.data&lt;scalar_t&gt;(),
            (unsigned long long*)mask.data&lt;int64_t&gt;());
      });

  at::Tensor mask_cpu = mask.to(at::kCPU);
  unsigned long long* mask_host = (unsigned long long*)mask_cpu.data&lt;int64_t&gt;();

  std::vector&lt;unsigned long long&gt; remv(col_blocks);
  memset(&amp;remv[0], 0, sizeof(unsigned long long) * col_blocks);

  at::Tensor keep =
      at::empty({boxes_num}, boxes.options().dtype(at::kLong).device(at::kCPU));
  int64_t* keep_out = keep.data&lt;int64_t&gt;();

  int num_to_keep = 0;
  for (int i = 0; i &lt; boxes_num; i++) {
    int nblock = i / threadsPerBlock;
    int inblock = i % threadsPerBlock;

    if (!(remv[nblock] &amp; (1ULL &lt;&lt; inblock))) {
      keep_out[num_to_keep++] = i;
      unsigned long long* p = mask_host + i * col_blocks;
      for (int j = nblock; j &lt; col_blocks; j++) {
        remv[j] |= p[j];
      }
    }
  }

  AT_CUDA_CHECK(cudaGetLastError());
  return
      order_t
          .index({keep.narrow(/*dim=*/0, /*start=*/0, /*length=*/num_to_keep)
                      .to(order_t.device(), keep.scalar_type())});
}

</code></pre>

username: tom
cooked: <p>Did you use torch.cuda.synchronize() before timing starts and ends?</p>
<p>Best regards</p>
<p>Thomas</p>

username: Ryan95
cooked: <p>how can i do it in libtorch?</p>

username: tom
cooked: <aside class="quote" data-post="2" data-topic="36763">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/any-analog-of-torch-cuda-syncronize-in-c-api/36763/2">Any analog of torch.cuda.syncronize() in C++ API?</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    Hi, 
You can call directly the cudaDeviceSynchronize() method as is done by the python code <a href="https://github.com/pytorch/pytorch/blob/202eaa4ef40824d08ae88b3062b9c82d78794432/torch/csrc/cuda/Module.cpp#L212-L218" rel="noopener nofollow ugc">here</a>.
  </blockquote>
</aside>