Problem for Init method on DDP

omidnr: <p>I started to learn Pytorch recently and I have to make a parallel process using pytorch DDP, i followed tutorials on the official website but I have a problem that I am facing a problem when I move from single GPU to Multi GPU. I use Jupyter Notebook server to write my code and I have direct access to physical server with GPUs (not a cloud service) i used torchrun to ease the things, here I modified the code to make it explicitly work on one GPU. I am also writing the original error here maybe someone can help me fix this, I think I have problems writing the INIT METHOD part or there should be a detail that I miss.</p>
<p>def setup_ddp():<br>
init_process_group(backend=“nccl”, init_method=“file:///tmp/myfile”)</p>
<p>class Trainer:<br>
def <strong>init</strong>(<br>
self,<br>
model: torch.nn.Module,<br>
train_data: DataLoader,<br>
optimizer: torch.optim.Optimizer,<br>
save_every: int,<br>
) → None:<br>
self.gpu_id = int(os.environ[“LOCAL_RANK”])<br>
self.model = model.to(self.gpu_id)<br>
self.train_data = train_data<br>
self.optimizer = optimizer<br>
self.save_every = save_every<br>
self.epochs_run = 0<br>
self.model = DDP(model, device_ids=[self.gpu_id])<br>
def main():</p>
<pre><code>    import os
    os.environ["WORLD_SIZE"] = "1"  # explicitly force to have just one gpu
        
    # torch.cuda.set_device(rank)
    # parameters
    total_epochs = 2
    save_every = 1
    batch_size = 32
    
    setup_ddp()
    dataset, model, optimizer = load_train_objs()
    train_data = prepare_dataloader(dataset, batch_size)        
    trainer = Trainer(model, train_data, optimizer, save_every)
    trainer.train(total_epochs)
    # destroy to clean up
    destroy_process_group()
    print("Checkpoint after Clean up")
</code></pre>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
main()</p>
<hr>
<h2><a name="error-1" class="anchor" href="#error-1"></a><strong>ERROR:</strong></h2>
<p>python DDP_torchrun.py<br>
Using device: cuda<br>
Using device: cuda<br>
Running DDP test on rank 0.<br>
Using device: cuda<br>
Running DDP test on rank 2.<br>
Checkpoint after setupddp<br>
Using device: cuda<br>
Running DDP test on rank 1.<br>
Checkpoint after setupddp<br>
Checkpoint after setupddp<br>
Files already downloaded and verified<br>
Files already downloaded and verified<br>
Files already downloaded and verified<br>
Checkpoint load train objs<br>
Checkpoint after train data<br>
Checkpoint load train objs<br>
Checkpoint after train data<br>
Checkpoint load train objs<br>
Checkpoint after train data<br>
d784bed9d024:9720:9720 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.2&lt;0&gt;<br>
d784bed9d024:9720:9720 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation<br>
d784bed9d024:9720:9720 [0] NCCL INFO cudaDriverVersion 12000<br>
NCCL version 2.19.3+cuda12.3<br>
d784bed9d024:9722:9722 [0] NCCL INFO cudaDriverVersion 12000<br>
d784bed9d024:9722:9722 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.2&lt;0&gt;<br>
d784bed9d024:9722:9722 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation<br>
d784bed9d024:9722:9780 [0] NCCL INFO Failed to open libibverbs.so[.1]<br>
d784bed9d024:9722:9780 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2&lt;0&gt;<br>
d784bed9d024:9722:9780 [0] NCCL INFO Using non-device net plugin version 0<br>
d784bed9d024:9722:9780 [0] NCCL INFO Using network Socket<br>
d784bed9d024:9720:9781 [0] NCCL INFO Failed to open libibverbs.so[.1]<br>
d784bed9d024:9720:9781 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2&lt;0&gt;<br>
d784bed9d024:9720:9781 [0] NCCL INFO Using non-device net plugin version 0<br>
d784bed9d024:9720:9781 [0] NCCL INFO Using network Socket<br>
d784bed9d024:9721:9721 [0] NCCL INFO cudaDriverVersion 12000<br>
d784bed9d024:9721:9721 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.2&lt;0&gt;<br>
d784bed9d024:9721:9721 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation<br>
d784bed9d024:9721:9782 [0] NCCL INFO Failed to open libibverbs.so[.1]<br>
d784bed9d024:9721:9782 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2&lt;0&gt;<br>
d784bed9d024:9721:9782 [0] NCCL INFO Using non-device net plugin version 0<br>
d784bed9d024:9721:9782 [0] NCCL INFO Using network Socket<br>
d784bed9d024:9721:9782 [0] NCCL INFO comm 0x563ed4aaac90 rank 1 nranks 3 cudaDev 0 nvmlDev 0 busId 18000 commId 0xe4aaadc79d89d14e - Init START<br>
d784bed9d024:9722:9780 [0] NCCL INFO comm 0x56271462c250 rank 2 nranks 3 cudaDev 0 nvmlDev 0 busId 18000 commId 0xe4aaadc79d89d14e - Init START<br>
d784bed9d024:9720:9781 [0] NCCL INFO comm 0x5569f44a8730 rank 0 nranks 3 cudaDev 0 nvmlDev 0 busId 18000 commId 0xe4aaadc79d89d14e - Init START</p>
<p>d784bed9d024:9722:9780 [0] init.cc:824 NCCL WARN Duplicate GPU detected : rank 2 and rank 0 both on CUDA device 18000</p>
<p>d784bed9d024:9720:9781 [0] init.cc:824 NCCL WARN Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 18000</p>
<p>d784bed9d024:9721:9782 [0] init.cc:824 NCCL WARN Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 18000<br>
d784bed9d024:9720:9781 [0] NCCL INFO init.cc:1396 → 5<br>
d784bed9d024:9722:9780 [0] NCCL INFO init.cc:1396 → 5<br>
d784bed9d024:9721:9782 [0] NCCL INFO init.cc:1396 → 5<br>
d784bed9d024:9722:9780 [0] NCCL INFO group.cc:64 → 5 [Async thread]<br>
d784bed9d024:9721:9782 [0] NCCL INFO group.cc:64 → 5 [Async thread]<br>
d784bed9d024:9720:9781 [0] NCCL INFO group.cc:64 → 5 [Async thread]<br>
d784bed9d024:9721:9721 [0] NCCL INFO group.cc:418 → 5<br>
d784bed9d024:9720:9720 [0] NCCL INFO group.cc:418 → 5<br>
d784bed9d024:9721:9721 [0] NCCL INFO group.cc:95 → 5<br>
d784bed9d024:9720:9720 [0] NCCL INFO group.cc:95 → 5<br>
d784bed9d024:9722:9722 [0] NCCL INFO group.cc:418 → 5<br>
d784bed9d024:9722:9722 [0] NCCL INFO group.cc:95 → 5<br>
Traceback (most recent call last):<br>
File “/home/omid/DDP_torchrun.py”, line 158, in <br>
mp.spawn(main, args=(world_size,), nprocs=n_gpus)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py”, line 241, in spawn<br>
return start_processes(fn, args, nprocs, join, daemon, start_method=“spawn”)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py”, line 197, in start_processes<br>
while not context.join():<br>
File “/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py”, line 158, in join<br>
raise ProcessRaisedException(msg, error_index, failed_process.pid)<br>
torch.multiprocessing.spawn.ProcessRaisedException:</p>
<p>– Process 1 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py”, line 68, in _wrap<br>
fn(i, *args)<br>
File “/home/omid/DDP_torchrun.py”, line 143, in main<br>
trainer = Trainer(model, train_data, optimizer, rank, world_size, save_every)<br>
File “/home/omid/DDP_torchrun.py”, line 65, in <strong>init</strong><br>
self.model = DDP(model.cuda(),device_ids=[rank]) # wrap the model with DDP       .cuda()<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py”, line 798, in <strong>init</strong><br>
_verify_param_shape_across_processes(self.process_group, parameters)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/distributed/utils.py”, line 263, in _verify_param_shape_across_processes<br>
return dist._verify_params_across_processes(process_group, tensors, logger)<br>
torch.distributed.DistBackendError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1691, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.19.3<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.<br>
Last error:<br>
Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 18000</p>