Confusing about distributed and muiltiprocessing

username: Kevinkevin189
cooked: <p>The multiprocessing and distributed confusing me a lot  when I’m reading some code</p>
<pre><code class="lang-python">#the main function to enter
def main_worker(rank,cfg):
    trainer=Train(rank,cfg)
if __name__=='_main__':
    torch.mp.spawn(main_worker,nprocs=cfg.gpus,args=(cfg,))

#here is a slice of Train class
class Train():
    def __init__(self,rank,cfg):
         #nothing special
         if cfg.dist:
#forget the indent problem cause I can't make it aligned
dist.init_process_group(backend='gloo',rank=self.rank,world_size=self.config.gpus,init_method="file://./sharefile",timeout=datetime.timedelta(seconds=5))
#init the model
self.model=torch.nn.Parallel.DistributedDataParallel(self.model,device_ids=[self.gpus])
</code></pre>
<p>Cause I’m new to the distributed coding,Here are sth I can’t understand.</p>
<p><code>1</code><br>
From my perspective, the spawn fcn start a job by initializing a group of procs with arg(rank,cfg), which means a groups of main_worker funtion. so every single main_worker,as a process, maintains a trainer object.But I see insde the trainer object, dist package also init process group to init a group of processes. What is the difference?where lies the true process?are there two hirearchy groups?<br>
<code>2</code><br>
the dist.init_process_method reports runtime errors.When I choose nccl as backend,it runs and waits until reporting (no nccl module),How to install an nccl module? And I changed it to ‘gloo’,It runs and reports a runtime error with no message,Thus it is not able to run.How to debug?<br>
<code>3</code><br>
The init_process_group receives an argument init_method, which I can hardly understand,I leave it None but it bugged a message  (No rendezvous handler for env://) ,I followed the tutorial to set environments variables MASTER_ADDR MASTER_PORT, it seems no effect.So I have to change it to ‘file://./sharefile’,what is this ?I totoally messed.<br>
I<br>
<code>4</code><br>
Now I’m running the code on my laptop,a single machine node with 1 GPU.But I need to train my model on a single machine node with multiple GPU cards.Is there any code to change?</p>
<p>Hope Someone can solve my problem or, maybe leave a tutorial or example,something comprehensive,not the simple sample.<br>
Thank u so much.</p>

username: Kevinkevin189
cooked: <p>I also copied the code sample from pytorch official tutorials</p>
<pre><code class="lang-auto">#!/usr/bin/env python
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def run(rank, size):
    """ Distributed function to be implemented later. """
    pass

def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)


if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>
<p>this tutorial link is here <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="noopener nofollow ugc">distributed_tutorial_link</a><br>
But, I still got error.<br>
And this code can only run a part  by setting the init_method = ‘file://./sharedfile’.<br>
I got the error</p>
<pre><code class="lang-auto">Process Process-1:
Traceback (most recent call last):
  File "C:\Anaconda3\envs\pt\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Anaconda3\envs\pt\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Kevin\Documents\Scripts\inpaint\test.py", line 13, in init_process
    dist.init_process_group('gloo',rank=rank,world_size=size,init_method='file://./sharedfile')
  File "C:\Anaconda3\envs\pt\lib\site-packages\torch\distributed\distributed_c10d.py", line 433, in init_process_group
    timeout=timeout)
  File "C:\Anaconda3\envs\pt\lib\site-packages\torch\distributed\distributed_c10d.py", line 508, in _new_process_group_helper
    timeout=timeout)
RuntimeError
Process Process-2:
Traceback (most recent call last):
  File "C:\Anaconda3\envs\pt\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Anaconda3\envs\pt\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Kevin\Documents\Scripts\inpaint\test.py", line 13, in init_process
    dist.init_process_group('gloo',rank=rank,world_size=size,init_method='file://./sharedfile')
  File "C:\Anaconda3\envs\pt\lib\site-packages\torch\distributed\distributed_c10d.py", line 433, in init_process_group
    timeout=timeout)
  File "C:\Anaconda3\envs\pt\lib\site-packages\torch\distributed\distributed_c10d.py", line 508, in _new_process_group_helper
    timeout=timeout)
RuntimeError

Process finished with exit code 0

</code></pre>
<p>Is there anything wrong with  this  code sample or my laptop?</p>

username: pritamdamania87
cooked: <aside class="quote no-group" data-username="Kevinkevin189" data-post="1" data-topic="104966">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevinkevin189/48/15435_2.png" class="avatar"> Kevinkevin189:</div>
<blockquote>
<p>From my perspective, the spawn fcn start a job by initializing a group of procs with arg(rank,cfg), which means a groups of main_worker funtion. so every single main_worker,as a process, maintains a trainer object.But I see insde the trainer object, dist package also init process group to init a group of processes. What is the difference?where lies the true process?are there two hirearchy groups?</p>
</blockquote>
</aside>
<p><code>torch.mp.spawn</code> spawns the actual processes, <code>init_process_group</code> doesn’t create any new processes but just initializes the distributed communication between spawned processes. For example if you spawn 4 processes using mp.spawn and call init_process_group on those 4 processes, <code>init_process_group</code> would ensure all 4 processes discover each other and now you can run collective calls like <code>allreduce</code> among those processes.</p>
<aside class="quote no-group" data-username="Kevinkevin189" data-post="1" data-topic="104966">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevinkevin189/48/15435_2.png" class="avatar"> Kevinkevin189:</div>
<blockquote>
<p>the dist.init_process_method reports runtime errors.When I choose nccl as backend,it runs and waits until reporting (no nccl module),How to install an nccl module?</p>
</blockquote>
</aside>
<p>You can find the NCCL installation guide here: <a href="https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html" class="inline-onebox" rel="noopener nofollow ugc">Installation Guide :: NVIDIA Deep Learning NCCL Documentation</a></p>
<aside class="quote no-group" data-username="Kevinkevin189" data-post="1" data-topic="104966">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevinkevin189/48/15435_2.png" class="avatar"> Kevinkevin189:</div>
<blockquote>
<p>The init_process_group receives an argument init_method, which I can hardly understand,I leave it None but it bugged a message (No rendezvous handler for env://) ,I followed the tutorial to set environments variables MASTER_ADDR MASTER_PORT, it seems no effect.So I have to change it to ‘file://./sharefile’,what is this ?I totoally messed.</p>
</blockquote>
</aside>
<p>This section of our docs goes over details of <code>init_method</code> and how to use it: <a href="https://pytorch.org/docs/stable/distributed.html#tcp-initialization" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.1 documentation</a>. In particular setting <code>MASTER_ADDR</code> and <code>MASTER_PORT</code> and leaving <code>init_method=None</code> should work.</p>
<aside class="quote no-group" data-username="Kevinkevin189" data-post="1" data-topic="104966">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevinkevin189/48/15435_2.png" class="avatar"> Kevinkevin189:</div>
<blockquote>
<p>Now I’m running the code on my laptop,a single machine node with 1 GPU.But I need to train my model on a single machine node with multiple GPU cards.Is there any code to change?</p>
</blockquote>
</aside>
<p>You would need some changes where you need to one process per GPU on your single machine node. You can use <code>mp.spawn</code> to create one process per GPU.</p>

username: pritamdamania87
cooked: <aside class="quote no-group" data-username="Kevinkevin189" data-post="2" data-topic="104966">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevinkevin189/48/15435_2.png" class="avatar"> Kevinkevin189:</div>
<blockquote>
<pre><code class="lang-auto">from torch.multiprocessing import Process

def run(rank, size):
    """ Distributed function to be implemented later. """
    pass

def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)


if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>
</blockquote>
</aside>
<p>For this code, what is the error that you encountered?</p>
<aside class="quote no-group" data-username="Kevinkevin189" data-post="2" data-topic="104966">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevinkevin189/48/15435_2.png" class="avatar"> Kevinkevin189:</div>
<blockquote>
<p>And this code can only run a part by setting the init_method = ‘file://./sharedfile’.</p>
</blockquote>
</aside>
<p>Can you try something like <code>file:///tmp/sharedfile</code> instead? It looks like the file:// scheme is not handling <code>./</code> correctly.</p>

username: Kevinkevin189
cooked: <p>Thanks for your replies.I now can understand the procs and distributed a little.</p>
<p>I found the true bug when run this tutor code on a linux machine.It seems the windows OS only support sharedfile inter-process communication method.And Because of the file permission issue,it reported a Runtime error with no error messege.It also reported a same Runtime error w/ a ‘Permission denied’ message.<br>
Hope official team would fix this ‘bug’ (cause an error with no explicit messege is really annoying)</p>

username: pritamdamania87
cooked: <p>I got an error as follows:</p>
<pre><code class="lang-auto">RuntimeError: open(/sharedfile): Permission denied
</code></pre>
<p>Doesn’t this error message clearly indicate the problem?</p>

username: Kevinkevin189
cooked: <p>Thanks!<br>
Cause I’ve got a blank message  on my laotop (Windows 10 Professional).And I’ve got it after I transfer my code to a linux OS machine.</p>