Send computation to a remote gpu

username: xerxex
cooked: <p>Is there a way to define a remote GPU device inside our local code?</p>
<p>For example:</p>
<pre><code class="lang-auto">local_cpu = torch.device('cpu')
remote_device = ... (?)

model = Model().to(remote_device)
...
inputs = inputs.to(remote_device)
outputs = model(inputs)
outputs = outputs.to(local_cpu)
</code></pre>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/xerxex">@xerxex</a>, there is a <code>torch.distributed.rpc</code> package for this purpose. Please refer to the following docs:</p>
<ol>
<li>API doc: <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener">https://pytorch.org/docs/master/rpc.html</a>
</li>
<li>Tutorial: <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_tutorial.html</a>
</li>
</ol>
<p>For now, we do not yet support creating remote device like <code>torch.device('worker1/cuda0')</code>, but this is on our roadmap and we plan to implement this as a sugar layer on top of RPC. Applications should be able to do the same thing using our raw RPC API.</p>

username: xerxex
cooked: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>, thanks for the links. Unfortunately, I could not run the minimal example from the documentation. can you please correct me to run this?</p>
<p>Let’s say I can not proceed without the return value so I need rpc_sync.<br>
I created two python scripts. The first one is:</p>
<pre><code class="lang-auto"># On worker 0:
import torch
import torch.distributed.rpc as rpc

rpc.init_rpc("worker0", rank=0, world_size=2)
ret = rpc.rpc_sync("worker1", torch.add, args=(torch.ones(2), 3))
rpc.shutdown()
</code></pre>
<p>and the second one is:</p>
<pre><code class="lang-auto"># On worker 1:
import torch.distributed.rpc as rpc
rpc.init_rpc("worker1", rank=1, world_size=2)
rpc.shutdown()
</code></pre>
<p>Then, when I execute the first script, I run into this error:</p>
<pre><code class="lang-auto">  File "process_0.py", line 4, in &lt;module&gt;
    rpc.init_rpc("worker0", rank=0, world_size=2)
  File "/data/anaconda3/lib/python3.7/site-packages/torch/distributed/rpc/__init__.py", line 60, in init_rpc
    init_method, rank=rank, world_size=world_size
  File "/data/anaconda3/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 48, in rendezvous
    raise RuntimeError("`url` must be a string. {}: {}".format(type(url), url))
RuntimeError: `url` must be a string. &lt;class 'NoneType'&gt;: None

</code></pre>

username: mrshenli
cooked: <p>Have you set the master address and port for Gloo ProcessGroup? Sth like:</p>
<pre><code class="lang-auto">    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'
</code></pre>

username: xerxex
cooked: <aside class="quote no-group quote-modified" data-username="mrshenli" data-post="4" data-topic="71739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>os.environ[‘MASTER_ADDR’] = ‘localhost’ os.environ[‘MASTER_PORT’] = ‘29500’</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/mrshenli">@mrshenli</a>, I added as you said, but still the same error.<br>
The error comming from this line:</p>
<pre><code class="lang-auto">rpc.init_rpc("worker0", rank=0, world_size=2)
</code></pre>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/xerxex">@xerxex</a>, which version of PyTorch are you using?</p>
<pre><code class="lang-auto">  File "/data/anaconda3/lib/python3.7/site-packages/torch/distributed/rpc/__init__.py", line 60, in init_rpc
    init_method, rank=rank, world_size=world_size
</code></pre>
<p>Given this line above, it does seem to be <a href="https://github.com/pytorch/pytorch/blob/v1.4.0/torch/distributed/rpc/__init__.py#L57-L67" rel="nofollow noopener">v1.4</a> nor the current <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/rpc/__init__.py#L60-L77" rel="nofollow noopener">master branch</a>.</p>
<p>RPC is only available since v1.4.</p>

username: xerxex
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a>, my version is 1.4.0a0+a5272cb</p>

username: mrshenli
cooked: <p>I see, that’s the commit prior to the official v1.4.0 release, and that why the code from the error message looks different from v1.4.0.</p>
<p>In <a href="https://github.com/pytorch/pytorch/blob/a5272cb6433efe0bdaf76bff428ce961538daa25/torch/distributed/rpc/__init__.py#L60" rel="nofollow noopener">the version you are using</a> (Nov 22nd, 2019), the <code>init_rpc</code> API takes an <code>init_method</code> arg, which you need to set. It is the same <code>init_method</code> as how you would call <code>init_process_group</code>.</p>
<p>It will be easier if you switch to official v1.4 or the current master.</p>

username: mrshenli
cooked: <p>Honestly, we wouldn’t recommend using versions prior to v1.4.0, the API and behavior of RPC package are only officially announced as experimental in v1.4.0. So, even if you can get around <code>init_rpc</code> using your current PyTorch version by setting <code>init_method</code>, you might run into other issues later.</p>