RuntimeError: RPC has not been initialized. Call torch.distributed.rpc.init_rpc first

EvanChristan123: <p>I am running the FedAvg simulation using the Pytorch RPC, but when I run it, the server side throw the errors. It seems that it is my coding problem, but I don’t know what is the problem. Here are some related code snippets:</p>
<pre><code class="lang-auto">.
.
.
    #Start training
    if args.rank == 0:
        for e in range(args.epoch):
            processes = []
            q = mp.Queue()
            print("Server's Epoch:"+str(e+1))
            weight = copy.deepcopy(model.state_dict())
            for r in range(args.world_size):
                p = mp.Process(
                    target=run_worker,
                    args=(
                        r,
                        model,
                        args.lr,
                        train_loader[r],
                        device,
                        args.epoch,
                        weight,
                        q))
                processes.append(p)
                p.start()

            for p in processes:
                p.join()
.
.
.
</code></pre>
<p>And for the function run_worker:</p>
<pre><code class="lang-auto">def run_worker(rank, model, lr, train_loader, device, epoch, weight, q):
    out_weight = rpc.rpc_sync(f"Worker{rank}", train, args=(rank, model, lr, train_loader, device, epoch, weight))
    q.put([rank, out_weight]) 
</code></pre>
<p>What is my main problem?</p>
<h3>
<a name="error-logs-1" class="anchor" href="#error-logs-1"></a>Error logs</h3>
<p>Server initialized!<br>
Server’s Epoch:1<br>
Process Process-1:<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.9/multiprocessing/process.py”, line 315, in _bootstrap<br>
self.run()<br>
File “/usr/lib/python3.9/multiprocessing/process.py”, line 108, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/pi/FYP/FedAvg_RPC.py”, line 96, in run_worker<br>
out_weight = rpc.rpc_sync(f"Worker{rank}“, train, args=(rank, model, lr, train_loader, device, epoch, weight))<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/rpc/api.py”, line 75, in wrapper<br>
raise RuntimeError(<br>
RuntimeError: RPC has not been initialized. Call torch.distributed.rpc.init_rpc first.<br>
Process Process-2:<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.9/multiprocessing/process.py”, line 315, in _bootstrap<br>
self.run()<br>
File “/usr/lib/python3.9/multiprocessing/process.py”, line 108, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/pi/FYP/FedAvg_RPC.py”, line 96, in run_worker<br>
out_weight = rpc.rpc_sync(f"Worker{rank}”, train, args=(rank, model, lr, train_loader, device, epoch, weight))<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/rpc/api.py”, line 75, in wrapper<br>
raise RuntimeError(<br>
RuntimeError: RPC has not been initialized. Call torch.distributed.rpc.init_rpc first.<br>
Process Process-3:<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.9/multiprocessing/process.py”, line 315, in _bootstrap<br>
self.run()<br>
File “/usr/lib/python3.9/multiprocessing/process.py”, line 108, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/pi/FYP/FedAvg_RPC.py”, line 96, in run_worker<br>
out_weight = rpc.rpc_sync(f"Worker{rank}", train, args=(rank, model, lr, train_loader, device, epoch, weight))<br>
File “/usr/local/lib/python3.9/dist-packages/torch/distributed/rpc/api.py”, line 75, in wrapper<br>
raise RuntimeError(<br>
RuntimeError: RPC has not been initialized. Call torch.distributed.rpc.init_rpc first.</p>
<h3>
<a name="minified-repro-2" class="anchor" href="#minified-repro-2"></a>Minified repro</h3>
<p><em>No response</em></p>
<h3>
<a name="versions-3" class="anchor" href="#versions-3"></a>Versions</h3>
<p>Collecting environment information…<br>
PyTorch version: 1.8.0a0+37c1f4a<br>
Is debug build: False<br>
CUDA used to build PyTorch: None<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Debian GNU/Linux 11 (bullseye) (aarch64)<br>
GCC version: (Debian 10.2.1-6) 10.2.1 20210110<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] (64-bit runtime)<br>
Python platform: Linux-5.15.84-v8±aarch64-with-glibc2.31<br>
Is CUDA available: False<br>
CUDA runtime version: No CUDA<br>
CUDA_MODULE_LOADING set to: N/A<br>
GPU models and configuration: No CUDA<br>
Nvidia driver version: No CUDA<br>
cuDNN version: No CUDA<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] mypy==0.812<br>
[pip3] mypy-extensions==0.4.3<br>
[pip3] numpy==1.19.5<br>
[pip3] torch==1.8.0a0+37c1f4a<br>
[pip3] torchvision==0.9.0a0+01dfa8e<br>
[conda] Could not collect</p>

wanchaol: <p>Replied in the issue too, can you initialize the RPC package first by calling into rpc_init?</p>

EvanChristan123: <p>Fixed after the posting, I shutdown the RPC on server too early.</p>