Free async request object

seliad: <p>Hi,<br>
I noticed that each async torch.distributed request object holds a pointer to the sent tensor,<br>
therefore the buffer memory is not freed in the sender process until we explicitly call wait() or is_completed().<br>
I am looking for a way to overcome this.<br>
any suggestions?</p>

mrshenli: <p>Looks like only <code>work</code> from Gloo and MPI ProcessGroup holds those tensors. Does it work if you do not hold that async work/request object in application? Gloo and MPI ProcessGroup both have a queue and a runLoop to hold those work objects alive until processed.</p>
<p>Curious, if you don’t call <code>wait()</code> how do you know when the communication is finished and safe to consume the output?</p>

seliad: <p>If we destroy async objects before completion we get<br>
“Attempted destruction of AsyncWork before work has completed, terminating the program.”</p>
<p>Therefore I must save in application all the sent request objects.</p>
<p>(Answering your question: I don’t consume the output of these (<code>isend</code>) messages)<br>
( of course, the receiver calls <code>wait()</code> before consuming)<br>
I’m using MPI (cuda-aware openmpi), with async p2p messages.</p>
<p>The goal is simple</p>
<ol>
<li>send tons of isend messages,</li>
<li>get memory freed automatically on each completion</li>
<li>finally, <code>wait()</code> on all the isends at the end of the program.</li>
</ol>

mrshenli: <p>I see. Only send/recv/recvAnysource APIs returns <code>AsyncWork</code>, and the <code>AsyncWork</code> is not stored in the queue. Other collectives use a different <code>WorkEntry</code> data structure, which are stored in the queue. Looks to me we should consolidate these APIs to implement the same behavior.</p>
<p><a class="mention" href="/u/teng-li">@teng-li</a> <a class="mention" href="/u/pietern">@pietern</a> Any reason for having different async work data structures for MPI ProcessGroup?</p>

mrshenli: <p>Created <a href="https://github.com/pytorch/pytorch/issues/36140" rel="nofollow noopener">an issue</a> to track this. In the mean time, does it work for you if you put the async work into a queue and launch a separate thread to wait and dequeue? The wait API does release GIL, so it shouldn’t be blocking the main thread. This will lead to a similar behavior if we consolidate send/recv with collective async work. It won’t be perfect, as it does not guarantee immediately free tensors when comm is done when an earlier send/recv finished later. A better solution would need to install callbacks to MPI thread, which requires larger revamp and I am not sure if MPI supports that.</p>

seliad: <blockquote>
<p>Only send/recv/recvAnysource APIs returns  <code>AsyncWork</code> and the  <code>AsyncWork</code>  is not stored in the queue</p>
</blockquote>
<p>You mean <code>isends</code> too right?</p>

mrshenli: <aside class="quote no-group" data-username="seliad" data-post="6" data-topic="75507">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seliad/48/16357_2.png" class="avatar"> seliad:</div>
<blockquote>
<p>You mean <code>isends</code> too right?</p>
</blockquote>
</aside>
<p>Oh, sorry, I meant the C++ send/recv/recvAnysource API. The <code>isend</code> API is Python only. Both send and isend call into the same C++ send API, the only difference is whether it waits on the work.</p>

seliad: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> I tried the solution with the cleaner thread and it doesn’t work:<br>
seems like the <code>wait()</code> in the cleaner thread stops the whole process.<br>
<a href="https://github.com/pytorch/pytorch/blob/945d7a7408891e25bc54a65015724f6d2de644e6/torch/lib/c10d/ProcessGroupMPI.cpp#L217" rel="nofollow noopener">I think this is were its at in code.</a><br>
I could only make it work with</p>
<pre><code class="lang-auto">while not r.is_completed():
    pass
</code></pre>
<p>but performance suffered a lot (~2x slowdown) compared my previous solution.</p>

Tianyi_Zhang: <p>Hi I want to follow up on this thread. What is the status of this feature? What is the best practice to free the requests now?</p>
<p><a class="mention" href="/u/seliad">@seliad</a> wonder what you choset to implement eventually?</p>

seliad: <p>I see the issue <a class="mention" href="/u/mrshenli">@mrshenli</a> opened is still open.</p>
<p>I ended up adding some python code to handle the freeing of buffers. Just keep the requests and occasionally check for completion and clean.</p>

mrshenli: <p>cc <a class="mention" href="/u/agolynski">@agolynski</a> for MPI and c10d</p>