Ease development by running computations on remote GPU

username: meandmymind
cooked: <p>Hello!</p>
<p>For development I use local machine with no GPU and have a remote machine with GPU.</p>
<p>I like to debug my code via IDE tools but also want to have access to gpu.</p>
<p>Using something a-la vscode over ssh is kinda slow so I want to run my scripts locally but send some computations to remote machine.</p>
<p>Ideal variant</p>
<pre><code class="lang-python"># pytorch will connect to remote machine and start a process for GPU computation there
rpc_init("server_addr")

# all computations with model.parameters() will automagically execute on remote machine
model = Linear(3, 1).to("remote-gpu")

data = [
    (Tensor([1, 2, 3]), 1),  # may call .to("remote-gpu") as well 
    (Tensor([4, 5, 6]), 2),  # not too bad
]

# data will be automagically sent to remote machine inside model.__call__()
# or it is already there if used Tensor.to("remote-gpu")
for (sample, label) in :
    result = model(sample)
    loss = compute_loss(label, result)
    # this is done on remote machine as well
    optimizer.step()
</code></pre>
<p>So I will run <code>python script.py</code> on local machine and use my local debugging tools and all the code will be run locally except somewhere deep tensor operations will do rpc calls to remote gpu to calculate and then execution will be on my machine again.</p>
<p>Is there an easy API in torch.distributed.rpc to achieve this? If not easy, how can achieve this with current API?</p>

username: KFrank
cooked: <p>Hi Lain!</p>
<aside class="quote no-group" data-username="meandmymind" data-post="1" data-topic="121002" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/meandmymind/48/38183_2.png" class="avatar"> meandmymind:</div>
<blockquote>
<p>I like to debug my code via IDE tools but also want to have access to gpu.</p>
<p>Using something a-la vscode over ssh is kinda slow</p>
</blockquote>
</aside>
<p>May I recommend that you run “emacs” on your remote machine?  If you<br>
don’t have an X Server on your local machine (or you feel that X Display<br>
is too slow), you can run “emacs -nw” in a terminal window.</p>
<p>Best.</p>
<p>K. Frank</p>

username: wayi
cooked: <p>There are two options:</p>
<ol>
<li>A higher-level API <code>RemoteModule</code> (recommended):<br>
<a href="https://pytorch.org/docs/master/rpc.html#remotemodule" class="inline-onebox" rel="noopener nofollow ugc">Distributed RPC Framework — PyTorch master documentation</a>
</li>
</ol>
<p>You don’t need to explicitly write any RPC. Instead, you need to override the <code>forward</code> method. When you construct this <code>nn.Module</code> like module, need to explicitly specify the device placement (on a remote GPU in your case). The input tensors will be automatically placed to the same cuda device.</p>
<p>Another example can be found here: <a href="https://deploy-preview-1514--pytorch-tutorials-preview.netlify.app/advanced/rpc_ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Combining Distributed DataParallel with Distributed RPC Framework — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
<ol start="2">
<li>A lower-level API <code>RRef</code>:<br>
<a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-reinforcement-learning-using-rpc-and-rref" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed RPC Framework — PyTorch Tutorials 1.8.1+cu102 documentation</a>
</li>
</ol>
<p>You need to write your own RPC, and call <code>to_here()</code> to run your remote module in the RPC.</p>

username: meandmymind
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a>, thanks for answering! But… I don’t like this solution for several reasons:</p>
<ol>
<li>
<p>I usually use neovim as an IDE and I tried running it on remote machine and connect via ssh. It was unbearably slow. Maybe upgrading my local connection speed would resolve this problem, I should try.</p>
</li>
<li>
<p>This way I have to either sync all my dev environment between two machines or migrate to remote machine fully. It’s not a convenient solution because I don’t want to store my personal projects-related code on remote machine.</p>
</li>
</ol>

username: meandmymind
cooked: <p>Also I don’t think TUI IDE over ssh is much better than VSCode over ssh. Maybe something like <code>mosh</code> can help as it is more terminal-application-friendly but last time I used it it messed with syntax highlighting <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:"></p>

username: meandmymind
cooked: <p><a class="mention" href="/u/wayi">@wayi</a> Thanks for answering! I guess your first option is what I need but I have several questions about it.</p>
<ol>
<li>
<p>Documentation on <code>RemoteModule</code> says <em>RemoteModule is not currently supported when using CUDA tensors</em>, but you said <em>tensors will be automatically placed to the same cuda device</em>. Am I missing something? If CUDA tensors are not supported now, where can I track progress on this?</p>
</li>
<li>
<p>I still need to spawn a remote pytorch process manually every time I start my local process, right? Is there a solution to create a long-living remote process that will consume messages from different local processes?</p>
</li>
</ol>
<p>If not, I can automate my local build process to do something like</p>
<pre><code class="lang-auto">ssh remote-host 'cd proj-dir; python remote-worker.py';
python train.py
</code></pre>
<p>It’s not very elegant but should work.</p>

username: wayi
cooked: <blockquote>
<ol>
<li>Documentation on <code>RemoteModule</code> says <em>RemoteModule is not currently supported when using CUDA tensors</em>, but you said <em>tensors will be automatically placed to the same cuda device</em>. Am I missing something? If CUDA tensors are not supported now, where can I track progress on this?</li>
</ol>
</blockquote>
<p>Thanks for pointing this out! The doc is outdated. Actually CUDA tensors are now supported on <strong>TensorPipe backend</strong>, documented on the same page. I will update the doc soon.</p>
<blockquote>
<ol start="2">
<li>I still need to spawn a remote pytorch process manually every time I start my local process, right? Is there a solution to create a long-living remote process that will consume messages from different local processes?</li>
</ol>
</blockquote>
<p>You have to initiate both local process(es) and remote workers together every time. This is because at the very beginning a static process group needs to be built, and the remote module(s) will be destroyed if the process group is gone.</p>
<p>What you are asking is more like a treating remote module as a server, and a local process can connect to that server whenever it needs to offload some work to the server. This can cause a problem – if multiple local processes offloads some work to the same remote worker, it will slow down the training.</p>
<p>RPC framework usually works in an opposite way: a local process can be viewed as a master process, and you will try to distribute different modules to different remote workers. Note that a remote module does not really have to be allocated to another machine – it can be on a different device of the same machine. The model parallelism idea is distributing different subsets of a module to different devices, which can be on the same machine or different machines. As a user, you shouldn’t feel any difference in the usage though.</p>

username: wayi
cooked: <blockquote>
<p>ssh remote-host ‘cd proj-dir; python remote-worker.py’;<br>
python train.py</p>
</blockquote>
<p>I am not sure this will work in your environment. You still need to make sure different hosts are connected, so you probably need to use sth like <strong>SLURM</strong> to deploy multi-node training.</p>

username: meandmymind
cooked: <aside class="quote no-group" data-username="wayi" data-post="7" data-topic="121002">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayi/48/32374_2.png" class="avatar"> wayi:</div>
<blockquote>
<p>does</p>
</blockquote>
</aside>
<p>Thanks for your answer! Now I start to see why it’s not that convenient to fulfill my use case with current framework solution.</p>
<p>For me I will have only one “main” and one “remote” worker so it’s not hard. But for general rpc it’s not good.</p>

username: wayi
cooked: <aside class="quote no-group" data-username="meandmymind" data-post="9" data-topic="121002">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/meandmymind/48/38183_2.png" class="avatar"> meandmymind:</div>
<blockquote>
<p>For me I will have only one “main” and one “remote” worker so it’s not hard. But for general rpc it’s not good.</p>
</blockquote>
</aside>
<p>RPC framework is mainly used for model parallelism. It seems that your use case is very different from this purpose.</p>
<p>Update: We have an ongoing project called elastic RPC, which should be able to work for your use case.</p>