Fetching data from remote server in dataloader

Kanishk_Jain: <p>I have a large hd5 file (~100GB) containing image features from resnet. This file is located on my local machine (laptop). My model is trained on cluster node that has storage limit of 25GB.<br>
Right now, I am using torch.distributed.rpc for tranferring data from my local machine to cluster.<br>
I am exposing a server on my local machine in the following way,</p>
<blockquote>
<p>num_worker = 4<br>
utils.WORLD_SIZE = num_worker + 1</p>
</blockquote>
<pre><code class="lang-auto">import os
import torch
import utils
import torch.distributed.rpc as rpc

def run_worker(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '8888'
    
    rpc.init_rpc(utils.SERVER_NAME,
                 rank = rank, 
                 world_size = world_size)
    print("Server Initialized", flush=True)
    rpc.shutdown()

if __name__ == "__main__":
    rank = 0
    world_size = utils.WORLD_SIZE
    run_worker(rank, world_size)
</code></pre>
<p>This server sends data from local machine to cluster. (other classes are omitted)</p>
<p>Now for requesting data from cluster, I am initializing rpc for each worker using <strong>worker_init_fn</strong> for dataloader,</p>
<pre><code class="lang-auto">def worker_init_fn(worker_id):
    rpc.init_rpc(utils.CLIENT_NAME.format(worker_id+1),
                rank=worker_id+1, world_size=utils.WORLD_SIZE)
    
    worker_info = torch.utils.data.get_worker_info()
    dataset = worker_info.dataset
    worker_id = worker_info.id
    server_info = rpc.get_worker_info(utils.SERVER_NAME)
    dataset.server_ref = rpc.remote(server_info, utils.Server)
</code></pre>
<p>Now, when I run my code, the training loop completes one iteration of the dataset and hangs after that and I get the following error on cluster side,</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "custom_datasets.py", line 134, in &lt;module&gt;
    main()
  File "custom_datasets.py", line 110, in main
    for i, (images, labels) in enumerate(mn_dataset_loader):
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 135, in _worker_loop
    init_fn(worker_id)
  File "custom_datasets.py", line 78, in worker_init_fn
    rank=worker_id+1, world_size=utils.WORLD_SIZE)
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/distributed/rpc/__init__.py", line 67, in init_rpc
    store, _, _ = next(rendezvous_iterator)
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 168, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: connect() timed out.
</code></pre>
<p>Above problem doesn’t occur when I set num_worker = 0, but the cluster code is very slow. I think the error is because of multi-threading but I am not sure how to resolve this. Please help me resolving the issue.</p>

mrshenli: <p>Hey <a class="mention" href="/u/kanishk_jain">@Kanishk_Jain</a></p>
<p>Sorry about the late response.</p>
<pre><code class="lang-auto">    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '8888'
</code></pre>
<p>If you are trying to use the distributed package across multiple machines, the <code>MASTER_ADDR</code> needs to be a routable IP, and <code>localhost</code> is not a routable one. One way to test if the specified IP works is to try if <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="nofollow noopener"><code>torch.distributed.init_process_group</code></a> can successfully run.</p>
<p>BTW, could you please post followup questions under the “distributed-rpc” tag? The PT Distributed team subscribes to updates with “distributed” and “distributed-rpc” tags to make sure to join the discussions promptly.</p>