Distributed Model Parallel Using Distributed RPC

Vibhatha_Abeykoon: <p>Hi,</p>
<p>I am a looking into the Distributed RPC API. In particular in this example</p>
<p><a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-rnn-using-distributed-autograd-and-distributed-optimizer" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-rnn-using-distributed-autograd-and-distributed-optimizer</a>.</p>
<p>I have a question when we have to scale the model into multiple machines.</p>
<p>Think of an instance where we have 16 layers (L1 — L16) where we have 4 machines each with 4 GPU devices. So my model will scale into all 16 GPUs in 4 machines such that each layer occupies the memory of a one GPU device. Assume a theoretical scenario memory is enough for computation.</p>
<p>When I need to do such a task, my training script must be written in such a way that if the original model was M, now I have M1 – M16 smaller models which depends upon the output of the previous model in the sequence.</p>
<p>I am not sure whether this is the best way to do this. If this is wrong, please explain the best practices with the RPC API.</p>
<p>Furthermore, here M1–M4 makes sense and that can be done by <code>to(device)</code> and when I have to send the input from M4 to M5, M4 is in GPU:3 of machine 1, M5 is in the GPU:0 of machine 2, I need to some how use an RPC call and send that data to the machine 2. This is the same case for all boundary conditions. Data could be sent some how via a synchronization mechanism.</p>
<p>Is this something possible with the existing APIs. I am not quite clear how DistributedOptimizer and Distributed Autograd could handle this.</p>
<p>These are some questions I have about the distributed model parallel.</p>
<p>Thank You,<br>
Vibhatha</p>

mrshenli: <p>Hey <a class="mention" href="/u/vibhatha_abeykoon">@Vibhatha_Abeykoon</a>, thanks for the question, this actually relates to several WIP projects that we are working on now.</p>
<blockquote>
<p>When I need to do such a task, my training script must be written in such a way that if the original model was M, now I have M1 – M16 smaller models which depends upon the output of the previous model in the sequence.</p>
</blockquote>
<p>In this case, you need 4 instead of 16 smaller models, and within each model you can use <code>Tensor.to(device)</code> to move data across GPUs as you mentioned below. For pipeline parallelism using RPC, <a href="https://github.com/pytorch/tutorials/pull/948/files" rel="noopener nofollow ugc">this tutorial</a> can serve as a reference (will be released with v1.6).</p>
<blockquote>
<p>I am not sure whether this is the best way to do this. If this is wrong, please explain the best practices with the RPC API.</p>
</blockquote>
<p>This is <strong>not</strong> the most convenient way to support pipeline parallelism. RPC is a lower-level API that offers flexibility but would require additional application code to orchestrate. One of the projects that we are looking into is to provide a higher-level API, e.g., a <code>DistributedPipelineParallel</code> (DPP) (similar to the <code>DistributedDataParallel</code>) which, ideally, can automatically divide the original model and place model shards (maybe) by using additional configuration hints or specific model structure (e.g., <code>nn.Sequential</code>). But this is still in discussion and no committed release date for it yet. Please do comment if you have suggestions or requirements on this feature.</p>
<blockquote>
<p>I need to some how use an RPC call and send that data to the machine 2. This is the same case for all boundary conditions.</p>
</blockquote>
<p>If you want the distributed autograd to automatically take care of the backward pass across machines, then yes, you will need to use RPC to send the intermediate output form machine 1 to machine 2. As of v1.6, RPC only accepts CPU tensors, so you will need to first move the tensor from <code>cuda:3</code> to <code>cpu</code> on machine 1 and then move the received tensor from <code>cpu</code> to <code>cuda:0</code> on machine 2. We explicitly added this restriction to avoid  unintentional device mapping errors through RPC. And we are working on a new device placement API (similar to the <code>map_location</code> in <code>torch.load</code>) to make this easier, where application can define default device mappings between each pair of nodes and directly pass GPU tensors to RPC. We hope we can get this done in v1.7.</p>
<blockquote>
<p>Data could be sent some how via a synchronization mechanism.</p>
</blockquote>
<p>What do you mean by “a synchronization mechanism” here?</p>
<blockquote>
<p>Is this something possible with the existing APIs. I am not quite clear how DistributedOptimizer and Distributed Autograd could handle this.</p>
</blockquote>
<p>Yep, the tutorial linked above shows an example.</p>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> This tutorial is wonderful. I was writing one and I think I can learn a lot from yours. I had questions when I was attempting this.</p>
<p>What I meant from synchronization is as the Model-Shard in Machine 1 needs to complete compute to start the Model-Shard2, even in pipeline case. Please correct me if I am wrong.<br>
So the Machine 2 Model-Shard2 must wait, until it gets the weights from Machine 2. And again, I am still going through your tutorial and may have answers in there.</p>
<p>One more thing, is how are we deploying this multi-machine model parallel module?<br>
Will there be modifications to torch.distributed.launch? What is the current method to launch this<br>
as we do for DDP?</p>
<p>The Pipeline parallel API would be just great. I was attempting to wrap this with Torchgpipe or Pipedream and I also felt like it is better if it can come within the PyTorch APIs.</p>
<p>I have a few suggestions, if we can make use of to(device) call into to(machine:device) kind of an API endpoint, it will be much easier to work, but I am not quite sure how the changes should reflect internally.</p>
<p>The DPP module needs a few features.</p>
<ol>
<li>How to partition the model, (a profiler based auto-partitioned or a manual one so that user can say how to partition). Model partition in a manual way and saying .to(device) is not going to work when we have to deal with complex and large models. So if this could be handled internally, it will be ideal for all users.</li>
<li>From the knowledge of using <a href="https://github.com/kakaobrain/torchgpipe" rel="nofollow noopener">Torchgpipe</a>, Rematerialization as a custom option within DPP would be a great option. There are couple reasons for this, some applications need performance using pipelining rather than saving memory. So in this case one could have the ability to turn it off and on depending on the training job. DPP could have a flag rematerialztion=False/True to make it disabled or enabled for the user.</li>
<li>With the <a href="https://github.com/msr-fiddle/pipedream" rel="nofollow noopener">Pipedream</a> work, it was very clear that the multi-machine involvement could be very useful for training and their profiler usage is important in getting a better DPP.</li>
<li>Enabling checkpointing internally for DPP would be easier as handling this manually could be troublesome. When the shards are distributed across machines the checkpoint itself needs to be a distributed entity (in my understanding).</li>
</ol>
<p>These could be great additions to DPP if possible.</p>
<p>Regrading the tutorial:</p>
<p><a href="https://github.com/pytorch/tutorials/blob/release/1.6/intermediate_source/dist_pipeline_parallel_tutorial.rst" rel="nofollow noopener">PP Tutorial</a></p>
<p>The <code>ResnetBase</code> and <code>self._lock</code> are not clear. Does the lock represents thread lock or something else?<br>
Is it possible to access the code for ResnetBase?</p>
<p>Thank You,<br>
Vibhatha</p>

mrshenli: <aside class="quote no-group" data-username="Vibhatha_Abeykoon" data-post="3" data-topic="87875">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar"> Vibhatha_Abeykoon:</div>
<blockquote>
<p>What I meant from synchronization is as the Model-Shard in Machine 1 needs to complete compute to start the Model-Shard2, even in pipeline case. Please correct me if I am wrong.</p>
</blockquote>
</aside>
<p>Yep, this is correct. That tutorial uses <code>RRef.to_here()</code> to block wait for the result. The downside is that this would block one RPC thread until <code>to_here()</code> returns. If this is a concern, the <code>async_execution</code> decorator can help. [<a href="https://github.com/pytorch/tutorials/pull/1045/files" rel="noopener nofollow ugc">tutorial</a>]</p>
<blockquote>
<p>One more thing, is how are we deploying this multi-machine model parallel module?<br>
Will there be modifications to torch.distributed.launch? What is the current method to launch this<br>
as we do for DDP?</p>
</blockquote>
<p>We don’t have a helper launching script for RPC yet as of v1.6. The RPC processes will need to be launched manually or programmably in application code. Added <a href="https://github.com/pytorch/pytorch/issues/40974" class="inline-onebox" rel="noopener nofollow ugc">Add a launching script for RPC · Issue #40974 · pytorch/pytorch · GitHub</a> to track.</p>
<blockquote>
<p>I have a few suggestions, if we can make use of to(device) call into to(machine:device) kind of an API endpoint, it will be much easier to work, but I am not quite sure how the changes should reflect internally.</p>
</blockquote>
<p>Right! This aligns with the remote device feature that we would love to build on top of RPC, but we don’t have bandwidth to cover that yet. It won’t be very hard to convert every operation of a remote device tensor into an RPC invocation, however that will be too slow due to the per-op comm overhead. Ideally, we should have a remote Tensor type that can do op fusing when possible, similar to lazy tensor. But as this would require a lot of effort and we haven’t seen too many requests for this yet, this feature didn’t make into our top priorities for now. We will come back to re-evaluate after the next release.</p>
<blockquote>
<p>How to partition the model, (a profiler based auto-partitioned or a manual one so that user can say how to partition). Model partition in a manual way and saying .to(device) is not going to work when we have to deal with complex and large models. So if this could be handled internally, it will be ideal for all users.</p>
</blockquote>
<blockquote>
<p>With the <a href="https://github.com/msr-fiddle/pipedream" rel="noopener nofollow ugc">Pipedream</a> work, it was very clear that the multi-machine involvement could be very useful for training and their profiler usage is important in getting a better DPP.</p>
</blockquote>
<p>Thanks a lot for all the suggestions!! Totally agree. Profiling is great and can definitely provide an easier entry point, especially when the application does not need to squeeze out the last bit of performance. For more perf-centric use cases, we might still want to allow users to hand-craft model partitioning and placement, maybe, by accepting some hints/configs.</p>
<blockquote>
<p>Enabling checkpointing internally for DPP would be easier as handling this manually could be troublesome. When the shards are distributed across machines the checkpoint itself needs to be a distributed entity (in my understanding).</p>
</blockquote>
<p>Exactly, this is a feature gap in RPC. We might be able to add checkpointing support to the WIP <a href="https://github.com/pytorch/pytorch/blob/00651b8c93354da13ab30c3fefd8cd529dc883e7/torch/distributed/nn/api/remote_module.py" rel="noopener nofollow ugc">RemoteModel</a> feature and build DPP on top.</p>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Thank you so much for this detailed explanation. I will try to design on top of what is offered from the PyTorch APIs. The plan you suggested is great and hope to use these in near future.</p>
<p>Regrading the tutorial:</p>
<p><a href="https://github.com/pytorch/tutorials/blob/release/1.6/intermediate_source/dist_pipeline_parallel_tutorial.rst" rel="nofollow noopener">PP Tutorial</a></p>
<p>The <code>ResnetBase</code> and <code>self._lock</code> are not clear. Does the lock represents thread lock or something else?<br>
Is it possible to access the code for ResnetBase?</p>
<p>Thank you,<br>
Vibhatha.</p>

mrshenli: <blockquote>
<p>The  <code>ResnetBase</code>  and  <code>self._lock</code>  are not clear. Does the lock represents thread lock or something else?</p>
</blockquote>
<p>Ah, thanks for the catch. Yep, this is a thread lock to prevent race. The full example code is here: <a href="https://github.com/pytorch/examples/blob/master/distributed/rpc/pipeline/main.py" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/master/distributed/rpc/pipeline/main.py</a></p>
<p>Let me add the missing <code>ResNetBase</code> to the tutorial.</p>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Thank you very much. I will try it.  <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>Just following up with you based on the performance factor.</p>
<p>For distributed model parallelism, could MPI-collective communication be a better choice than distributed RPC? I mean these are two different models designed to serve two different purposes. But, at the end of the day what we would be doing is sending or receiving data from one point to another point. In terms of performance does PyTorch Distributed RPC outperforms MPI-Collectives (especially ISend/IRecv, Send/Recv)? Is this something PyTorch community already considered and decided to go with RPCs instead of MPI libraries?</p>
<p>But I understand the currently distributed optimizer, Autograd and those extensible components have been written to support RPC. But does MPI stand a chance here?</p>

mrshenli: <p>Hey <a class="mention" href="/u/vibhatha_abeykoon">@Vibhatha_Abeykoon</a></p>
<p>We will announce a new RPC backend in v1.6, which is called <a href="https://pytorch.org/docs/master/rpc.html#tensorpipe-backend" rel="nofollow noopener">TensorPipe</a>. This is a P2P comm library and is designed to automatically figure out the best comm media between two RPC workers, e.g., shm, tcp, nvlink, ib, etc. (This is still WIP) We will gradually make TensorPipe the default RPC backend and retire ProcessGroup RPC backend due to the perf reasons as you noticed. The original reason for adding ProcessGroup RPC backend is to have a working comm module to unblock other parts of the system, and also buy us time to design better solutions.</p>
<p>Regarding MPI, we probably will not develop an RPC backend on top of MPI but we do welcome OSS contributions or it might be possible to add MPI as a channel type in TensorPipe. One downside of using MPI is that there are different implementations does not seem to be one implementation that rules all use cases. That’s also one reason why PyTorch does not include MPI as a submodule but require the users to provide a MPI installation and compile from source.</p>
<p>Is there any specific reason for requesting MPI RPC backend?</p>

Vibhatha_Abeykoon: <p>Hey <a class="mention" href="/u/mrshenli">@mrshenli</a>.</p>
<p>The main reason is that there are tons of <strong>Scientific Applications written on MPI</strong> and it would be really hard to port them back to a different backend. These applications will do an MPI_Init somewhere at the very beginning of the program. In the early part of the program, there are specific scientific data pre-processing, shallow/complex algorithms applied to pre-process the data. Then comes the DL workload. Such applications are very common in the high-performance computing domain. So supporting MPI-backend could be very vital to support such applications seamlessly without breaking the data pipeline/training.</p>
<p>I understand there are many MPI-implementations. But MPI still can be left to the user to install, but the specifications are mostly consistent in most of the MPI implementations. All it needs is a wrapper library to wrap the collective communication calls. PyTorch already has this API in C10D. Please correct me if I am wrong.</p>
<p>Tensorpipe seems to be a very interesting project that could glue all these together.</p>

mrshenli: <p>I see. Technically, it shouldn’t be too hard to let the ProcessGroup RPC backend to use MPI, as it only requires its send/recv features. One options could be adding one field to ProcessGroup RPC backend construction time options, and let users decide whether they want to use Gloo, or NCCL (&gt; 2.7), or MPI.</p>
<p>cc <a class="mention" href="/u/lcw">@lcw</a> any thoughts on MPI + TensorPipe?  Does it make sense to add MPI as a channel type for TensorPipe?</p>

lcw: <p>I don’t understand the argument for using MPI in RPC: what does the fact that other libraries use the MPI API have to do with the RPC library using it under the hood? AFAIK, MPI is not incompatible with Gloo or TensorPipe: the same process can use them both, in different parts of the code. Also, the fact that RPC uses MPI internally does not help with porting MPI code to RPC: it would still have to be rewritten to use the RPC interface.</p>
<p>A good reason would be if there was a difference in performance. Have you reason to believe there is?</p>
<p>If we were stuck with the ProcessGroup agent only, then I could agree that we should allow it to use MPI instead of Gloo, but as it’s slated to go away in favor of the TensorPipe-based one this change may not end up being so useful. TensorPipe is natively asynchronous, and thus suits really well the use-case of RPC, contrary to Gloo and MPI which are blocking. We have already proven that TensorPipe outperforms Gloo for RPC. It may be different for MPI, as some MPI implementations use specialized backends, but that’s what TensorPipe is also going to do.</p>

lcw: <p>My two cents on the above discussion too: ideally you shouldn’t specialize your code to handle differently the transfers between GPUs on the same node and between different nodes. By doing so you couple your code with your deployment, meaning you need to rewrite some parts to change from 4 GPUs/host to single-GPU hosts and so on. With the TensorPipe agent you will be able to perform RPC calls between GPUs on a node and the data will still be transferred over NVLink just as if you had done t.to(…). So with no performance overhead you get code that is resilient to a topology change.</p>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/lcw">@lcw</a> It is not an argument, just asking if this is something possible with the current implementations you have. With MPI asynchronous calls you can still get asynchronous attribute to the code.</p>
<p><a href="https://www.mpich.org/static/docs/latest/www3/MPI_Isend.html" rel="noopener nofollow ugc">ISend</a>, <a href="https://www.mpich.org/static/docs/latest/www3/MPI_Irecv.html" rel="noopener nofollow ugc">IRecv</a></p>
<p>Correct me if I am wrong.</p>
<aside class="quote no-group" data-username="lcw" data-post="12" data-topic="87875">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b5e925/48.png" class="avatar"> lcw:</div>
<blockquote>
<p>Also, the fact that RPC uses MPI internally does not help with porting MPI code to RPC: it would still have to be rewritten to use the RPC interface.</p>
</blockquote>
</aside>
<p>Yes, we have to still re-write, but unless MPI backend support is there, the communication on RPC channels will have different performance. Have you benchmarked the performance of Gloo, TensorPipe RPC vs MPI? The reason for asking the MPI compatibility is that, a program would not only have a training script. It has a data pre-processing functions, training and post-processing based on the trained model. If a system designed with an MPI backend is used to be as the main framework where PyTorch act as a library within the code, in those cases the support from MPI is immensely important. The use cases of using PyTorch is getting complex and complex, I think that is why a library like Tensorpipe is also coming into play.</p>
<p>I just wanted to point out a possible usage of MPI within the distribution.</p>
<aside class="quote no-group" data-username="lcw" data-post="13" data-topic="87875">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b5e925/48.png" class="avatar"> lcw:</div>
<blockquote>
<p>With the TensorPipe agent you will be able to perform RPC calls between GPUs on a node and the data will still be transferred over NVLink just as if you had done t.to(…). So with no performance overhead you get code that is resilient to a topology chang</p>
</blockquote>
</aside>
<p>This is really useful for model parallelism and writing complex networks with feedback loops.</p>
<p>Is Tensor-pipe going to be a standalone library or is this going to be adopted in <code>torch.distributed.rpc</code> ?</p>

iffiX: <p>MPI is not appropriate for serving as a new backend of RPC.<br>
Their model are totally different and inherently incompatible with each other.</p>
<p>Now the serious explanation, to put it simply:</p>
<p>MPI style is tightly coupled, P2P primitives like recv, send, irecv, isend are there simply because you wouldn’t want to introduce an additional library to complete a simple P2P commnication, like collecting a state or log.</p>
<p>RPC style is complete decoupled, services are there and you can access it if a process want and have the permission to. Therefore synchronization becomes a disaster because processes are <strong>distributed</strong>.</p>
<p>Tensorpipe is mainly just a smart payload delivery layer, it is there because the important “decision” feature will greatly improve the performance of rpc, since unoptimized rpc libraries such as “GRPC”, “Dubbo” “Swift” does not handle tensors on defferent devices well. It is designed for the distributed scenario. And it can also handle elasticity and dynamic size (Eg: initialize your program with different number of process-roles, esbecially important if you want to add some springness to your application) very well, in this case, MPI is just <strong>way too rigid</strong>.</p>
<p>BTW, distributed applications are <strong>complex</strong>, you cannot expect pytorch to be any simpler because it is already very simple, its rpc api could be considered “overly simple and crude” if you compare it to an industrial grade rpc framework “Dubbo” designed by Alibaba:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bc0cea37218ca28d3fba1017f3a24d958fb4b9b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1bc0cea37218ca28d3fba1017f3a24d958fb4b9b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bc0cea37218ca28d3fba1017f3a24d958fb4b9b_2_667x500.jpeg" alt="image" data-base62-sha1="3XvYHYNDY8MPVFKKzBCCaJqiKNl" width="667" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bc0cea37218ca28d3fba1017f3a24d958fb4b9b_2_667x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bc0cea37218ca28d3fba1017f3a24d958fb4b9b.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bc0cea37218ca28d3fba1017f3a24d958fb4b9b.jpeg 2x" data-dominant-color="C4D8CC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">900×674 255 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, in a word, please, don’t mix these two things together, they are different.</p>

iffiX: <p>You can use MPI to implement rpc, techincally, but the performance could be really bad, for example, in order to send a message of arbitrary length, in MPI you need to send the size to your target, then the target has to allocate the memory, then you can send the payload, since MPI is not a raw connection like tcp or infiniband, you would expect more delay in these two communications, and you have to deal with process failures! MPI will fail if any component process has failed, and that’s why we would like to remove that behavior in rpc, see <a href="https://discuss.pytorch.org/t/how-to-catch-exceptions-caused-by-rpc-exactly/88856/10">88856</a>.</p>

ptrblck: <p><a class="mention" href="/u/iffix">@iffiX</a> I would like to ask you to keep the discussion objective.<br>
If you disagree with something, explain your position and keep the discussion alive.<br>
While the majority of your post is a great explanation, the first part is unfortunately not. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

iffiX: <p>Inappropriate apart removed and updated, sorry for any issues cased by the provoking part in the comment. also <a class="mention" href="/u/vibhatha_abeykoon">@Vibhatha_Abeykoon</a></p>

iffiX: <p><a class="mention" href="/u/vibhatha_abeykoon">@Vibhatha_Abeykoon</a> <a class="mention" href="/u/mrshenli">@mrshenli</a><br>
I am reviewing this topic today and I have a few suggestions.</p>
<p>I will use “process/device” (Eg: “worker:0/cuda:0”) as a location descriptor of a tensor, and tensor is the only holder of all data.</p>
<p>The first thing is:</p>
<aside class="quote no-group" data-username="Vibhatha_Abeykoon" data-post="3" data-topic="87875">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar"> Vibhatha_Abeykoon:</div>
<blockquote>
<p>How to partition the model</p>
</blockquote>
</aside>
<p>I have a primitive design for this purpose: an <a>assigner</a> and a simple <a>wrapper</a>, assigner currently won’t partition a model auto matically, instead it will just assign user specified partitions based on a series of heuristics (GPU mem, GPU power, CPU mem, CPU power, Model complexity, bandwidth of connection between models), but it could be also reworked to fulfill your purpose, and you just need to wrap all of your submodules in the wrapper, the wrapper just stores input/output location descritors, nothing more.</p>
<p>Simply speaking, partitioning just require users to specify the input and output (process/device) requirements for a module/model, and then a smart assigner to assign partitions to process/device.</p>
<p>Theoretically a dynamic profiler is much better than a static heuristic assigner, since it actively detects hotspots and try to even the load on all of your nodes, but this introduces additional issues: Is evening the load across nodes increasing the performance? How much cost have the additional transmission introduced? Is the decreasing the load not pushing your GPUs to their full capacity (kernel launching cost should be considered)? So there are possibilities that this solution does not meet with the theoretical standard.</p>
<p>I beilieve <a class="mention" href="/u/mrshenli">@mrshenli</a> has studied about this issue, from his <a href="https://mrshenli.github.io/" rel="noopener nofollow ugc">profile page</a>. I think many more experiments are needed to determine the best scheme.</p>
<p>The second thing is:</p>
<aside class="quote no-group" data-username="Vibhatha_Abeykoon" data-post="3" data-topic="87875">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar"> Vibhatha_Abeykoon:</div>
<blockquote>
<p>I have a few suggestions, if we can make use of to(device) call into to(machine:device) kind of an API endpoint, it will be much easier to work, but I am not quite sure how the changes should reflect internally.</p>
</blockquote>
</aside>
<p>I think it won’t be too defficult if <code>rpc.pair</code> in <a href="https://github.com/pytorch/pytorch/issues/41546" rel="noopener nofollow ugc">#41546</a> is implemented,</p>
<pre><code class="lang-auto">tensor.to("worker:1/cuda:0")
</code></pre>
<p>is equivalent to</p>
<pre><code class="lang-auto"># suppose there is a tensor on process "worker:0" and device "cuda:0" of this process
# move to process "worker:1" and device "cuda:1" of that process
# take care when torch.cuda.set_device is used

def pair_and_move_to(tensor, device, uuid):
    # uuid should be a unique identifier to identify this tensor
    # could be process_name:tensor_ptr
    tensor = tensor.to(device)
    rpc.pair(uuid, tensor)
    return RRef(tensor)

# on worker:0 when .to is invoked:
rpc.rpc_sync("worker:1", rpc.pair, args=(tensor, "cuda:1", tensor.some_uuid))
</code></pre>
<p>And for implementing <code>Distributed Model Parallel Using Distributed RPC</code>, there are many model parallel methods, it depends on your model, algorithm framework and application. For DDP compatible RPC solutions specifically, I have implementations of a <a href="https://github.com/iffiX/machin/blob/master/machin/parallel/server/param_server.py" rel="noopener nofollow ugc">gradient reduction server</a> in my framework, which should be able to do the exact same thing as DDP does, however, this server implementation is also based on new API RFC <a href="https://github.com/pytorch/pytorch/issues/41546" rel="noopener nofollow ugc">#41546</a>, which haven’t been implemented in pytorch. I have made a simple <a href="https://github.com/iffiX/machin/blob/master/machin/parallel/distributed/world.py" rel="noopener nofollow ugc">wrapper</a> upon current primitive RPC APIs for this RFC, it is not efficient since two primitive RPC requests have to be made per wrapped high level RPC API, but it is tested and workable, if you would like to take a look.</p>
<p>From my <em>personal</em> point of view, if torch could provide a way to “expose” a resource or service upon the RPC module, even <a href="https://github.com/pytorch/pytorch/blob/00651b8c93354da13ab30c3fefd8cd529dc883e7/torch/distributed/nn/api/remote_module.py" rel="noopener nofollow ugc">RemoteModule</a> could be easily implemented, since it is basically create a module on a remote process, then expose its “__call__()” method as a service on the global scope,  <a href="https://github.com/pytorch/pytorch/issues/41546" rel="noopener nofollow ugc">#41546</a> could solve this problem.</p>
<p>Summary:<br>
You can achieve all functions, using current torch APIs, if you don’t mind 20% ~ 30% efficiency loss, and spend a little?(much) time to construct your wheel, if you don’t, you can also use mine <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20">. It would be definetly better if torch could just provide these functions, with more optimizations.</p>
<p>And <a class="mention" href="/u/mrshenli">@mrshenli</a> <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a>, please chime in and offer some feedback and precious suggestions on  <a href="https://github.com/pytorch/pytorch/issues/41546" rel="noopener nofollow ugc">#41546</a>, there are some torchelastic issues I am not very familiar with and need some help <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

iffiX: <p>Oh, and about:</p>
<aside class="quote no-group" data-username="Vibhatha_Abeykoon" data-post="3" data-topic="87875">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar"> Vibhatha_Abeykoon:</div>
<blockquote>
<p>deploying this multi-machine model parallel module?<br>
Will there be modifications to torch.distributed.launch? What is the current method to launch this<br>
as we do for DDP?</p>
</blockquote>
</aside>
<p>That’s complex, my personal experience says that you should try to split and group your process functionalities, for example, grouping them by “role”:</p>
<p>(Image from RFC <a href="https://github.com/pytorch/pytorch/issues/41425" rel="noopener nofollow ugc">#41425</a>)<br>
          <a href="https://user-images.githubusercontent.com/43595115/87468085-cf977a80-c5cd-11ea-9be9-35cb3c580523.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://user-images.githubusercontent.com/43595115/87468085-cf977a80-c5cd-11ea-9be9-35cb3c580523.png" width="690" height="173">
          </a>
</p>
<p>This idea comes from “micro services”. It makes your application logic much more clearer to understand. RFC proposal <a href="https://github.com/pytorch/pytorch/issues/41546" rel="noopener nofollow ugc">#41546</a> also contains an automatic role-based launcher implementation to address this issue.</p>
<p>However, role based design is not compatiable with:</p>
<pre><code class="lang-auto">if __name__ == "__main__":
    ...
    tensor.to("worker:0/cuda:1")
    # do some computation
    tensor.to("worker:1/cuda:0")
</code></pre>
<p>because you are manually specifying every destination and location,</p>