Get dependency for a graph

Ruiyang_Chen: <p>Hi,</p>
<p>I’m trying to optimize the performance of distributed machine learning by assigning priorities for data packets. I’m wondering is there a way to get the dependency graph for computation/communication operations? In Tensorflow, we can extract the prerequisites for each tensor, but Pytorch seems to build the computation graph at the runtime and there’s no API to do so.</p>

mrshenli: <p><a class="mention" href="/u/ruiyang_chen">@Ruiyang_Chen</a></p>
<p>That’s a great question! This is actually one item on our roadmap.</p>
<blockquote>
<p>I’m wondering is there a way to get the dependency graph for computation/communication operations?</p>
</blockquote>
<ol>
<li>It is possible. One option is to traverse the autograd graph from the output. <a href="https://github.com/pytorch/pytorch/blob/f326045b3757236aabe367dfca1894be14ce31ef/torch/csrc/distributed/c10d/reducer.cpp#L588-L608" rel="noopener nofollow ugc">This code</a> shows how DDP reducer implements this currently.</li>
<li>Another option is to do sth similar to <a href="https://github.com/NVIDIA/apex" rel="noopener nofollow ugc">APEX</a> by logging the grad ready order in the first iteration, and use that to prioritize comms.</li>
<li>Comm (async allreduce) operations are kicked off whenever a gradient bucket is ready.</li>
</ol>
<p><a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">This</a> is a brief description of how DDP works. You can start from there. <a href="https://i.cs.hku.hk/~cwu/papers/yhpeng-sosp19.pdf" rel="noopener nofollow ugc">This paper</a> seems did sth similar for PyTorch.</p>
<p>Do you mind if I ask the scope of the project you are working on? The PyTorch Distributed team hasn’t started working on this yet, but we have some ideas regarding how to do it. If you plan to modify PyTorch code or open source your solution, we can collaborate on this effort.</p>

Ruiyang_Chen: <aside class="quote no-group" data-username="mrshenli" data-post="2" data-topic="73311">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<ol>
<li>kicked off whenever a gradient bucket is ready.</li>
</ol>
<p><a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">This </a> is a brief description of how DDP works. You</p>
</blockquote>
</aside>
<p>Hi Shen,</p>
<p>Thanks a lot for your reply! The scope of my work is similar to this paper TicTac: Accelerating Distributed Deep Learning with Communication Scheduling, <a href="https://arxiv.org/abs/1803.03288" class="inline-onebox" rel="noopener nofollow ugc">[1803.03288] TicTac: Accelerating Distributed Deep Learning with Communication Scheduling</a> that will use heuristics to pipelining communication and computation. Specifically, I want to design a new parameter server algorithm that will use a set of distributed servers and pipeline the communication/computation to improve the bottleneck.</p>

mrshenli: <p>For parameter server, <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener">torchrpc</a> might be helpful. It’s currently an experimental feature and various improvements will come soon.</p>