Tensor to cpu operation is bottleneck in forward pass

Pranjal_Agarwal: <p>I am using scipy’s <em>linear_sum_assignment</em> after my forward pass, before computing loss. (as done in DeTR - object detection with transformers’ github <a href="https://github.com/facebookresearch/detr/blob/eb9f7e03ed8e2ed2cd55528989fe7df890bc3fc0/models/matcher.py" rel="noopener nofollow ugc">codebase</a>.<br>
To use the scipy function, tensor needs to be transferred to cpu. That operation is taking a significant time (as expected), in between training. Is there a recommended good practice to transfer tensor to cpu, to optimise the overall time taken?<br>
In DeTR 's usage of <em>cpu()</em> (above), they have put the <strong>torch.no_grad()</strong> decorator over the function comprising <em>cpu()</em> , I tried doing the same, but did not observe any improvement in time consumption.</p>

KFrank: <p>Hi Pranjal!</p>
<aside class="quote no-group" data-username="Pranjal_Agarwal" data-post="1" data-topic="169432" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pranjal_agarwal/48/56094_2.png" class="avatar"> Pranjal_Agarwal:</div>
<blockquote>
<p>To use the scipy function, tensor needs to be transferred to cpu. That operation is taking a significant time (as expected),</p>
</blockquote>
</aside>
<p>I think that this is the best you can do.  To use scipy, as you recognize,<br>
your tensor must be on the cpu and it takes unavoidable time to move<br>
it there.</p>
<p>Nonetheless, some comments:</p>
<p>You could hypothetically implement “linear_sum_assignment” yourself<br>
on the gpu.  (This would almost certainly not be worth the considerable<br>
effort.)  Even if the gpu were not particularly well adapted to that algorithm,<br>
it probably wouldn’t be dramatically slower than the cpu, and you would<br>
save the non-trivial time of moving tensors back and forth.</p>
<p>You could hypothetically have your gpu performing some other useful<br>
computation internally while it’s moving the tensor to the cpu.  But guessing<br>
that your workflow looks like forward pass, linear_sum_assignment, loss<br>
computation, backward pass, I don’t see what other useful work would<br>
be available for your gpu during the gpu / cpu data transfer.</p>
<p>If your model (and / or input data and batch size) is <em>very</em> small, so that you<br>
aren’t really getting significant benefit from the gpu, running your entire<br>
training loop on the cpu could be faster.  But this would seem unlikely for<br>
any realistic use case.</p>
<p>Good luck.</p>
<p>K. Frank</p>

nwn: <p>Hello Pranjal,<br>
I totally agree with <a class="mention" href="/u/kfrank">@KFrank</a>, usually it is not worth to recode a function like that in PyTorch. But if you want to, take a look at <a href="https://github.com/scipy/scipy/blob/main/scipy/optimize/_lsap.c" rel="noopener nofollow ugc">LSAP</a>. If you want to see a C or C++ “unhidden” implementation I would search for “Jonker-Volgenant”.<br>
Best,<br>
nwn</p>

ytwanghaoyu: <p>Also, you can check this for a GPU + CPU  implementation of LAP using auction algorithm,</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/bkj/auction-lap">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/bkj/auction-lap" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/c7128d4d4924a745c8735a2b2fdf9315ef9a7c89072085e9228190dcfabbb362/bkj/auction-lap" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/bkj/auction-lap" target="_blank" rel="noopener nofollow ugc">GitHub - bkj/auction-lap: Auction algorithm for solving linear assignment...</a></h3>

  <p>Auction algorithm for solving linear assignment problem (LAP) - bkj/auction-lap</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>More LAP implementation can be found at:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/cheind/py-lapsolver">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/cheind/py-lapsolver" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/527bf37258ac82dafde7d860ba4574696c2af6889b9eec682de1238426d01cf4/cheind/py-lapsolver" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/cheind/py-lapsolver" target="_blank" rel="noopener nofollow ugc">GitHub - cheind/py-lapsolver: Fast linear assignment problem (LAP) solvers...</a></h3>

  <p>Fast linear assignment problem (LAP) solvers for Python based on c-extensions - cheind/py-lapsolver</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
and</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/berhane/LAP-solvers">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/berhane/LAP-solvers" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/4e53712a8736662e3ed71eb597cea525fdadabd156740063b51f6149945bdb2c/berhane/LAP-solvers" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/berhane/LAP-solvers" target="_blank" rel="noopener nofollow ugc">GitHub - berhane/LAP-solvers: Benchmarking Linear Assignment Problem Solvers</a></h3>

  <p>Benchmarking Linear Assignment Problem Solvers. Contribute to berhane/LAP-solvers development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>