The results is different when placing rpc_aync at a different .py file

username: cindybrain
cooked: <p>I want to execute a function at the worker side and return the results to the master. However, I find that the results is different when placing rpc_async at a different .py file</p>
<h4>
<a name="method-1-1" class="anchor" href="#method-1-1"></a>Method 1</h4>
<p>master.py:</p>
<pre><code class="lang-python">
import os
import torch
import torch.distributed.rpc as rpc
from torch.distributed.rpc import RRef
from test import sub_fun

os.environ['MASTER_ADDR'] = '10.5.26.19'
os.environ['MASTER_PORT'] = '5677'

rpc.init_rpc("master", rank=0, world_size=2)
rref = torch.Tensor([0])
sub_fun(rref)
rpc.shutdown()
</code></pre>
<p>test.py</p>
<pre><code class="lang-python">
def f(rref):
   print("function is executed on master")

def sub_fun(rref):
   x = rpc.rpc_async("worker", f, args=(rref,))
</code></pre>
<p>worker.py:</p>
<pre><code class="lang-python">import os
import torch
import torch.distributed.rpc as rpc
from torch.distributed.rpc import RRef

os.environ['MASTER_ADDR'] = '10.5.26.19'
os.environ['MASTER_PORT'] = '5677'

def f(rref):
    print("function is executed on worker")
rpc.init_rpc("worker", rank=1, world_size=2)
rpc.shutdown()
</code></pre>
<p>I found that the output is “function is executed on master” at the worker side.</p>
<h4>
<a name="method-2-2" class="anchor" href="#method-2-2"></a>Method 2</h4>
<p>when I put the two functions: sub_fun and f in the master.py rather than the test.py, the result is “function is executed on worker”.</p>
<p>Why the two ways output the different results. and how can I get the result 2 with the method 1.</p>
<p>My torch version is ‘1.5.0+cu92’</p>

username: H-Huang
cooked: <p>The outputs differ based on where you put <code>sub_fun</code> due to the order in which python will load the pickled function. RPC will pickle based on function name not based on function definition. If you want to keep both function implementations they should be renamed to something like <code>f_worker</code> and <code>f_master</code></p>
<p>For your case, to get the results you want, you can format your files like this:</p>
<p>master.py</p>
<pre><code class="lang-python">import os
import torch
import torch.distributed.rpc as rpc
from torch.distributed.rpc import RRef
from test import sub_fun

os.environ['MASTER_ADDR'] = '10.5.26.19'
os.environ['MASTER_PORT'] = '5677'

if __name__ == "__main__":
   rpc.init_rpc("master", rank=0, world_size=2)
   rref = torch.Tensor([0])
   sub_fun(rref)
   rpc.shutdown()
</code></pre>
<p>test.py</p>
<pre><code class="lang-python">import torch.distributed.rpc as rpc
from worker import f

def sub_fun(rref):
    x = rpc.rpc_async("worker", f, args=(rref,))
</code></pre>
<p>worker.py</p>
<pre><code class="lang-python">import os
import torch
import torch.distributed.rpc as rpc
from torch.distributed.rpc import RRef

os.environ['MASTER_ADDR'] = '10.5.26.19'
os.environ['MASTER_PORT'] = '5677'

def f(rref):
    print("function is executed on worker")

if __name__ == "__main__":
    rpc.init_rpc("worker", rank=1, world_size=2)
    rpc.shutdown()
</code></pre>

username: cindybrain
cooked: <p>Really thanks for your reply!<br>
However, I found it deadlock if I imported the function “f” from the worker.py. Did your code worked fine?</p>