Distributed pagerank with pytorch

username: ph0123
cooked: <p>I am trying to implement Pagerank with libtorch. I finished the OpenMPI version with Pagerank. I try to read libtorch documents <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="nofollow noopener">here</a>.</p>
<p>However, I did not see any function like RPC in OpenMPI.</p>
<p>Is there possible to implement Pagerank with libtorch?</p>

username: mrshenli
cooked: <p>It should be possible. And there are several levels of APIs that you can use:</p>
<ol>
<li>send/recv APIs: <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.send" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html#torch.distributed.send</a>
</li>
<li>collective communication APIs: <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce</a>
</li>
<li>RPC APIs:<br>
a. <a href="https://pytorch.org/docs/stable/rpc.html" rel="nofollow noopener">https://pytorch.org/docs/stable/rpc.html</a><br>
b. <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_tutorial.html</a><br>
c. <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html</a>
</li>
</ol>

username: ph0123
cooked: <p>Thank you so much!<br>
I will try.</p>

username: ph0123
cooked: <p>Hi,<br>
I try to use RPC, but it seem to difficult.<br>
My idea is that rank0 will call my_add function, and rank1 will do it and return the value for rank0.</p>
<pre><code class="lang-auto">"""RPC with Torch"""
"""run.py:"""
#!/usr/bin/env python
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process
import torch.distributed.rpc as rpc

def my_add(a,b):
    return a+b

def run(rank, size):
    tensor = torch.zeros(1)
    if rank == 0:
        rpc.init_rpc("worker0", rank=0, world_size=size)
        ret = rpc.rpc_sync("worker1", my_add, args=(2,3))
        print(ret)
    else:
        rpc.init_rpc("worker1", rank=1, world_size=size)
    rpc.shutdown()
    #print('Rank ', rank, ' has data ', tensor[0])

def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)


if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>
<p>I try to run but the error is as below</p>
<pre><code class="lang-auto">Process Process-45:
Process Process-46:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "&lt;ipython-input-22-0e6d083442bd&gt;", line 29, in init_process
    fn(rank, size)
  File "&lt;ipython-input-22-0e6d083442bd&gt;", line 29, in init_process
    fn(rank, size)
  File "&lt;ipython-input-22-0e6d083442bd&gt;", line 16, in run
    rpc.init_rpc("worker0", rank=0, world_size=size)
  File "&lt;ipython-input-22-0e6d083442bd&gt;", line 20, in run
    rpc.init_rpc("worker1", rank=1, world_size=size)
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/site-packages/torch/distributed/rpc/__init__.py", line 77, in init_rpc
    store, _, _ = next(rendezvous_iterator)
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/site-packages/torch/distributed/rpc/__init__.py", line 88, in init_rpc
    api._init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 172, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 283, in _init_rpc_backend
    rpc_backend_options=rpc_backend_options,
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/site-packages/torch/distributed/rpc/backend_registry.py", line 75, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
RuntimeError: Address already in use
  File "/Users/cnphuong/opt/anaconda3/lib/python3.7/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in _process_group_init_backend_handler
    "Default process group must not be initialized before init_rpc."
RuntimeError: Default process group must not be initialized before init_rpc.
</code></pre>
<p>Please help!<br>
Thanks.</p>

username: mrshenli
cooked: <blockquote>
<p>RuntimeError: Default process group must not be initialized before <code>init_rpc</code>.</p>
</blockquote>
<p>As the error suggested, <code>init_process_group</code> cannot be called before <code>init_rpc</code>, as currently DDP and RPC do not work together. We are working on dropping this requirement: <a href="https://github.com/pytorch/pytorch/issues/33583" class="inline-onebox" rel="noopener nofollow ugc">Avoid modifying default process group in init_rpc · Issue #33583 · pytorch/pytorch · GitHub</a></p>
<p>For the code snippet above, removing <code>dist.init_process_group(backend, rank=rank, world_size=size)</code> should work.</p>

username: ph0123
cooked: <p>Thank you so much!<br>
On MPI, I can use distributed object to remote other workers do something with other worker’s distributed object.<br>
On pytorch, I saw that it has <strong>Remote Reference (RRef)</strong>. However, I did not see how to create distributed object with Pytorch. Could you please suggest in this case?</p>
<p>For example,<br>
worker1 holds a dictionary with key and value DIC1.<br>
worker0 will send an array KEY_SENDs with the keys to worker1. Worker1 will check DIC1 and return an array with the values from KEY_SENDs and DIC1.</p>
<p>I try with</p>
<pre><code class="lang-auto">def get_values(dic1, arr):
    rest =  np.array([])
    for i in arr:
        rest= np.append(rest,dic1[i])
    return rest
        

def run(rank, size):
    tensor = torch.zeros(1)
    my_name = "worker"+str(rank)
    if rank == 0:
        thisdict =	{4:2,2:6,3:8}
        rpc.init_rpc(my_name,rank=0, world_size=size)
        target = 1
        target_name = "worker"+str(target)
        ret = rpc.rpc_sync(target_name, get_values, args=(thisdict,[1,2]))
        print(str(ret) + " is in rank 0 from rank 1")
    else:
        thisdict =	{2:1,1:3,4:2}
        rpc.init_rpc(my_name,rank=rank, world_size=size)
    rpc.shutdown()
    print(my_name)

</code></pre>
<p>I run it but no results were show on this example.</p>
<p>Thanks,</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/ph0123">@ph0123</a></p>
<p>I am not sure if I fully understand the use case. I created an example to show how to do intersect of a local dict and a remote dict. It shouldn’t too hard to extend this to do, e.g., a union or just sending keys. Let me know if this answers the question.</p>
<pre><code class="lang-python">import torch
import torch.distributed.rpc as rpc
import torch.multiprocessing as mp
import os


def create_dict():
    return {1:'a', 2:'b', 3:'c'}


def intersect_dict(dict1, dict2_rref):
    ret = {}
    for key in dict2_rref.local_value():
        if key in dict1:
            ret[key] = dict1[key]

    return ret


def run(dst):
    dict1 = {1:'a', 3:'c', 4:'d'}
    dict2_rref = rpc.remote(dst, create_dict)
    intersect = rpc.rpc_sync(dst, intersect_dict, args=(dict1, dict2_rref))
    print(intersect)


def run_worker(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'
    if rank == 1:
        rpc.init_rpc("worker0", rank=rank, world_size=world_size)
        run("worker1")
    else:
        rpc.init_rpc("worker1", rank=rank, world_size=world_size)

    rpc.shutdown()


if __name__=="__main__":
    world_size = 2
    mp.spawn(run_worker, args=(world_size, ), nprocs=world_size, join=True)
</code></pre>

username: ph0123
cooked: <p>Hi,<br>
That’s so good.<br>
However, I try to run your code with my PC. the output is</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
&lt;ipython-input-1-c234795e743f&gt; in &lt;module&gt;
     37     world_size = 2
     38 
---&gt; 39     mp.spawn(run_worker, args=(world_size, ), nprocs=world_size, join=True)

~/opt/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in spawn(fn, args, nprocs, join, daemon, start_method)
    198                ' torch.multiprocessing.start_process(...)' % start_method)
    199         warnings.warn(msg)
--&gt; 200     return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')

~/opt/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)
    156 
    157     # Loop on join until it returns True or raises an exception.
--&gt; 158     while not context.join():
    159         pass
    160 

~/opt/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    111                 raise Exception(
    112                     "process %d terminated with exit code %d" %
--&gt; 113                     (error_index, exitcode)
    114                 )
    115 

Exception: process 1 terminated with exit code 1
</code></pre>
<p>I try with</p>
<pre><code class="lang-auto">"""RPC with Torch"""
"""run.py:"""
#!/usr/bin/env python
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process
import torch.distributed.rpc as rpc
import numpy as np

def create_dict():
    return {1:'a', 2:'b', 3:'c'}


def intersect_dict(dict1, dict2_rref):
    ret = {}
    for key in dict2_rref.local_value():
        if key in dict1:
            ret[key] = dict1[key]
    return ret

def run(dst):
    dict1 = {1:'a', 3:'c', 4:'d'}
    dict2_rref = rpc.remote(dst, create_dict)
    intersect = rpc.rpc_sync(dst, intersect_dict, args=(dict1, dict2_rref))
    print(intersect)
        
def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    #dist.init_process_group(backend, rank=rank, world_size=size)
    #fn(rank, size)
    if rank == 1:
        rpc.init_rpc("worker0", rank=rank, world_size=world_size)
        fn("worker1")
    else:
        rpc.init_rpc("worker1", rank=rank, world_size=world_size)

    rpc.shutdown()

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>
<p>And it worked on my PC.</p>
<p>Could you please explain why mp.spawn error in this case?</p>
<p>thanks,</p>

username: ph0123
cooked: <p>Can I create the dict2 next to rpc.init_rpc as</p>
<pre><code class="lang-auto">if rank == 1:
        rpc.init_rpc("worker0", rank=rank, world_size=world_size)
        fn("worker1")
    else:
        # initial dict2 here.
        rpc.init_rpc("worker1", rank=rank, world_size=world_size)
</code></pre>
<p>My target is that each rank will hold their dictionary, and this dictionary is always available in the program. In your code, If I repeat the run() function, The dict1 is initial again.</p>
<p>For example, rank 0 call run function on rank 1. Your codes is good. But I wanna run it several time, and after calling function on rank 1. The rank 1 changes something in the dictionary. Then, in the next iteration, rank0 wanna get the value from updated dictionary on rank1.</p>
<p>Thanks,</p>

username: mrshenli
cooked: <p>That’s weird, I don’t know why <code>mp.spawn</code> would fail. Is it because it has to be <code>127.0.0.1</code> instead of <code>localhost</code> in your env? We can try add more logs/try-except to identify which line crashed.</p>
<blockquote>
<p>My target is that each rank will hold their dictionary, and this dictionary is always available in the program. In your code, If I repeat the run() function, The dict1 is initial again.</p>
</blockquote>
<p>If this is the only concern, you can move that <code>rpc.remote</code> call to <code>create_dict()</code> out of the <code>run</code> function and do it before the training loop?</p>
<blockquote>
<p>For example, rank 0 call run function on rank 1. Your codes is good. But I wanna run it several time, and after calling function on rank 1. The rank 1 changes something in the dictionary. Then, in the next iteration, rank0 wanna get the value from updated dictionary on rank1.</p>
</blockquote>
<p>Sure, there are many ways to solve this. For example, you can define the dict as a global value so that each process will have its own copy. Then in <code>intersect_dict</code>, just read from that global value instead of passing the <code>RRef</code> around.</p>
<p>Or, you can have a master that tells all workers to initialize their states upfront and then get <code>RRef</code>s of those states as return values.</p>
<p>If you need to construct an <code>RRef</code> from that dict, you can also use <code>rpc.RRef(dict_obj)</code> to create a local <code>RRef</code> and then pass that around through RPC.</p>

username: ph0123
cooked: <p>Hi,<br>
Thank you so much!<br>
I search the documents, but I did not know how to make a dictionary as a global value with PyTorch.<br>
I worked with MPI and UPC++. MPI provides a <strong>distributed object</strong>, which helps each worker to hold their values. UPC++ also provides <strong>distributed objects</strong> that are a similar name on all workers but different memory locations. Additionally, UPC++ creates <strong>Global Pointers</strong> to remote from other workers. I quit easy.</p>
<p>However, I did not know these kinds of variables in Pytorch documents.<br>
Do you have any suggestion?</p>
<p>I have another question. If i run with several iterations, is there any function to waiting all workers as barrier() function.</p>
<p>Thanks.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/ph0123">@ph0123</a></p>
<p>I was referring to Python global vars, sth like:</p>
<pre><code class="lang-python">import torch
import torch.distributed.rpc as rpc
import torch.multiprocessing as mp
import os


_local_dict = {}


def intersect_dict(dict1):
    ret = {}
    for key in _local_dict:
        if key in dict1:
            ret[key] = dict1[key]

    return ret


def run(dst):
    intersect = rpc.rpc_sync(dst, intersect_dict, args=(_local_dict,))
    print(intersect)


def run_worker(rank, world_size):
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    if rank == 0:
        _local_dict.update({1:'a', 3:'c', 4:'d'})
        rpc.init_rpc("worker0", rank=rank, world_size=world_size)
        run("worker1")
    else:
        _local_dict.update({1:'a', 2:'b', 3:'c'})
        rpc.init_rpc("worker1", rank=rank, world_size=world_size)

    rpc.shutdown()


if __name__=="__main__":
    world_size = 2
    mp.spawn(run_worker, args=(world_size, ), nprocs=world_size, join=True)
</code></pre>

username: ph0123
cooked: <p>Wow. It worked.<br>
Thank you for your supports.</p>

username: ph0123
cooked: <p>Hi,</p>
<p>I try with the loop.<br>
After each iteration, I used <code>dist.barrier()</code>, and the program is not stop because of waiting others rpc.</p>
<p>What is the problem with it?</p>
<p>Thanks,</p>

username: mrshenli
cooked: <blockquote>
<p>the program is not stop because of waiting others rpc.</p>
</blockquote>
<p>What does this mean by “the program is not stop”? If you are referring to the behavior that some RPC requests are still running in background, then yes, this is the expected behavior. Because RPC server has its own thread pool to handle requests, and <code>dist.barrier</code> only blocks the thread that runs it. Why do you want to combine <code>dist.barrier()</code> with RPC? Is this just to conclude an iteration?</p>

username: ph0123
cooked: <p>HI,<br>
I did not know this is not work with RPC.<br>
Because of my algorithm, step (k+1) is depended on step k.<br>
I want to make sure that all worker finished before starting new iteration.</p>
<blockquote>
<p>each worker hold their own dictionary.<br>
step k:<br>
call rpc to other workers to check and update dictionary values, which is based on value on step (k-1).<br>
<span class="hashtag-raw">#I</span> put barrier here<br>
step k+1:<br>
call rpc to other workers to check and update dictionary values, which is based on value on step (k).<br>
…</p>
</blockquote>
<p>with RPC pytorch, Could you please sugges any function like dis.barrier()?</p>
<p>Thanks,</p>

username: mrshenli
cooked: <p>Hey,</p>
<p><code>dist.barrier()</code> and RPC should both work as expected. The contract <code>dist.barrier()</code> offers is only blocking until all participating threads reach the same barrier. So, as RPC threads in background are not calling <code>dist.barrier</code>, they are not part of that synchronization.</p>
<p>If you would like to synchronize all RPCs, one option is doing sth similar to <code>_wait_all_workers</code> as linked below. It uses rank 0 as a coordinator to tell all others when to proceed.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/479b04e26a59d20b72ad5fdaec819caaad49af75/torch/distributed/rpc/api.py#L137-L199" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/479b04e26a59d20b72ad5fdaec819caaad49af75/torch/distributed/rpc/api.py#L137-L199" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/479b04e26a59d20b72ad5fdaec819caaad49af75/torch/distributed/rpc/api.py#L137-L199</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="137" style="counter-reset: li-counter 136 ;">
<li>@_require_initialized</li>
<li>def _wait_all_workers():</li>
<li>    r"""</li>
<li>    Block until all local and remote RPC processes reach this method and wait</li>
<li>    for all outstanding work to complete. Every RPC process must call this</li>
<li>    method before exit to perform a graceful shutdown. This should be used to</li>
<li>    terminate the RPC framework, and there is no guarantee that the RPC</li>
<li>    framework will work after this method returns.</li>
<li>    """</li>
<li>    assert (</li>
<li>        _ALL_WORKER_NAMES is not None</li>
<li>    ), "`_ALL_WORKER_NAMES` is not initialized for `def _wait_all_workers`."</li>
<li>    leader_worker_name = sorted(_ALL_WORKER_NAMES)[0]</li>
<li>
</li><li>    self_worker_name = _get_current_rpc_agent().get_worker_info().name</li>
<li>
</li><li>    global _wait_all_workers_sequence_id</li>
<li>    with _wait_all_workers_dict_lock:</li>
<li>        sequence_id = _wait_all_workers_sequence_id</li>
<li>        _wait_all_workers_sequence_id += 1</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/479b04e26a59d20b72ad5fdaec819caaad49af75/torch/distributed/rpc/api.py#L137-L199" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>In general, all we need for a synchronization is to join background threads into the current thread that calls <code>dist.barrier</code>. For example, the following code won’t work, as <code>some_func</code> is running on a different thread.</p>
<pre><code class="lang-python">rpc.rpc_async(to, some_func)
dist.barrier()
</code></pre>
<p>However, the code below should work, as it blocks the current thread before calling the <code>barrier</code></p>
<pre><code class="lang-python">fut = rpc.rpc_async(to, some_func)
fut.wait()
dist.barrier()
</code></pre>
<p>So, as a summary, if you would like to use the <code>dist.barrier</code> to synchronize RPCs, the application will need to collect the futures created in one iteration, and wait for all of them to complete before calling <code>barrier</code>.</p>

username: ph0123
cooked: <p>Hi,<br>
I got it.<br>
Thank you so much!</p>