Initialize DDP with torch.distributed.run/torchrun

shrutishrestha: <p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#initialize-ddp-with-torch-distributed-run-torchrun:~:text=%2C%20world_size" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.12.0+cu102 documentation</a>)-,Initialize%20DDP%20with%20torch.distributed.run/torchrun,-We%20can%20leverage</p>
<p>I am trying to tun this DDP example but I am getting this error:<br>
ValueError: The hostname of the rendezvous endpoint ‘:29400’ must be a dot-separated list of labels, an IPv4 address, or an IPv6 address.</p>

ptrblck: <p>I guess you haven’t set the <code>MASTER_ADDR</code> as described in the tutorial?</p>

shrutishrestha: <p>The earlier problem was resolved, but got a  new problem<br>
while setting gloo in init_method… it stucks in loss.backward and produces the below error:<br>
RuntimeError: […/third_party/gloo/gloo/transport/tcp/pair.cc:589] Read error [10.245.10.159]:15026: Connection reset by peer</p>
<p>while setting nccl in init_method… it stucks in<br>
ddp_model = DistributedDataParallel(model, device_ids=[local_rank])<br>
and produces<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3</p>
<p>ncclSystemError: System call (socket, malloc, munmap, etc) failed.</p>
<pre><code class="lang-auto">        dist.init_process_group("nccl", init_method='env://', world_size=size, rank=rank)
        gpu = torch.device("cuda",local_rank)

        if args["model"] == "srcnn":
            model = SRCNN.Net().to(gpu)

        ddp_model = DistributedDataParallel(model, device_ids=[local_rank])
        
        #initializing checkpoint
        best_val_metric = 0
        checkpoint_file = args["srcnn_x2"]    
        
        map_location = {'cuda:%d' % 0: 'cuda:%d' % local_rank}

        state_dict = torch.load(checkpoint_file, map_location=map_location)
        model.load_state_dict(state_dict)

        # initializing loss, optimizer, scheduler s
        batch_size = args["train_batch_size"]
        batch_size_per_gpu = batch_size // size
        criterion = torch.nn.MSELoss(reduction='mean').cuda(gpu)
        optimizer = optim.Adam(ddp_model.parameters(), lr=0.01)
        scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=4)

        traindataset = FitsDataset(args, data_list = trainpathlist)     
        valdataset = FitsDataset(args, data_list = valpathlist)  
        
        train_sampler = torch.utils.data.distributed.DistributedSampler(traindataset,
                                                                        num_replicas= size,
                                                                        rank= rank)
        val_sampler = torch.utils.data.distributed.DistributedSampler(valdataset,
                                                                        num_replicas= size,
                                                                        rank= rank)
        
        trainLoader = DataLoader(traindataset, batch_size=args["train_batch_size"], shuffle=False,       
                                     num_workers=0, pin_memory=True, sampler=train_sampler)
        
        valLoader = DataLoader(valdataset, batch_size=args["val_batch_size"], shuffle=False, 
                                   num_workers=0, pin_memory=True, sampler=val_sampler)
        
        best_ckpt = {"epoch":-1, "current_val_metric":0, "model":ddp_model.state_dict()}
        epoch = args["start_iters"]

        if  rank == 0: start = datetime.now()         
        total_step = len(trainLoader)

        for epoch in range(args["max_epochs"]):
            
            if  rank == 0: start_dataload = time()
            total_step = len(trainLoader)

            for i, ( image, label) in enumerate(trainLoader):
                image = image.to(torch.float32)
                label = label.to(torch.float32)
                
                image = image.to(gpu, non_blocking=True)
                label = label.to(gpu, non_blocking=True)

                if  rank == 0: stop_dataload = time()

                if  rank == 0: start_training = time()

                optimizer.zero_grad()

                #forward pass
                output = ddp_model(image)
                loss = criterion(output, label)

                #backward and optimize
                loss.backward()
                optimizer.step()

                if  rank == 0: stop_training = time() 
                if (i + 1) % 200 == 0 and  rank == 0:
                    print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Time data load: {:.3f}ms, Time training: {:.3f}ms'.format(epoch + 1, args.epochs,i + 1, total_step, loss.item(), (stop_dataload - start_dataload)*1000,(stop_training - start_training)*1000))  

                if  rank == 0: start_dataload = time()

            #Save checkpoint at every end of epoch
            if  rank == 0:
                torch.save(ddp_model.state_dict(), './checkpoint/{}GPU_{}epoch.checkpoint'.format( size, epoch+1))

        if  rank == 0:
            print("&gt;&gt;&gt; Training complete in: " + str(datetime.now() - start))
</code></pre>

ptrblck: <p>Could you rerun the NCCL use case via <code>export NCCL_DEBUG=INFO</code> and post the logs here?</p>

shrutishrestha: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="156744">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>export NCCL_DEBUG=INFO</p>
</blockquote>
</aside>
<p>Traceback (most recent call last):<br>
File “arctic_run_folders/basecode_multinode_multigpu.py”, line 248, in <br>
Traceback (most recent call last):<br>
File “arctic_run_folders/basecode_multinode_multigpu.py”, line 248, in <br>
Traceback (most recent call last):<br>
File “arctic_run_folders/basecode_multinode_multigpu.py”, line 248, in <br>
Traceback (most recent call last):<br>
File “arctic_run_folders/basecode_multinode_multigpu.py”, line 248, in <br>
main(args, rank, local_rank, size, cpus_per_task, hostnames, gpu_ids, NODE_ID, MASTER_ADDR)<br>
main(args, rank, local_rank, size, cpus_per_task, hostnames, gpu_ids, NODE_ID, MASTER_ADDR)<br>
File “arctic_run_folders/basecode_multinode_multigpu.py”, line 46, in main<br>
File “arctic_run_folders/basecode_multinode_multigpu.py”, line 46, in main<br>
main(args, rank, local_rank, size, cpus_per_task, hostnames, gpu_ids, NODE_ID, MASTER_ADDR)<br>
File “arctic_run_folders/basecode_multinode_multigpu.py”, line 46, in main<br>
ddp_model = DistributedDataParallel(model, device_ids=[local_rank])<br>
File “/userapp/virtualenv/SR_ENV/venv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 578, in <strong>init</strong><br>
main(args, rank, local_rank, size, cpus_per_task, hostnames, gpu_ids, NODE_ID, MASTER_ADDR)<br>
File “arctic_run_folders/basecode_multinode_multigpu.py”, line 46, in main<br>
ddp_model = DistributedDataParallel(model, device_ids=[local_rank])<br>
File “/userapp/virtualenv/SR_ENV/venv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 578, in <strong>init</strong><br>
ddp_model = DistributedDataParallel(model, device_ids=[local_rank])<br>
File “/userapp/virtualenv/SR_ENV/venv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 578, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.<br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.<br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.<br>
ddp_model = DistributedDataParallel(model, device_ids=[local_rank])<br>
File “/userapp/virtualenv/SR_ENV/venv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py”, line 578, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.<br>
srun: error: acidsgcn001: tasks 0-1: Exited with exit code 1<br>
srun: error: acidsgcn002: tasks 2-3: Exited with exit code 1</p>

shrutishrestha: <p>My jobsubmit file is</p>
<p>#!/bin/bash<br>
<span class="hashtag">#SBATCH</span> --job-name=pytorch_multinode<br>
<span class="hashtag">#SBATCH</span> -w acidsgcn001,acidsgcn002<br>
<span class="hashtag">#SBATCH</span> --nodes=2<br>
<span class="hashtag">#SBATCH</span> --ntasks-per-node=2<br>
<span class="hashtag">#SBATCH</span> --gres=gpu:2<br>
<span class="hashtag">#SBATCH</span> --cpus-per-task=4<br>
<span class="hashtag">#SBATCH</span> --partition=qGPU48<br>
<span class="hashtag">#SBATCH</span> --mem=128gb<br>
<span class="hashtag">#SBATCH</span> --gres=gpu:V100:2<br>
<span class="hashtag">#SBATCH</span> --time=01:00:00<br>
<span class="hashtag">#SBATCH</span> --mail-type=END,BEGIN,FAIL<br>
<span class="hashtag">#SBATCH</span> --mail-user=sshrestha8@student.gsu.edu<br>
<span class="hashtag">#SBATCH</span> --account=csc344r73<br>
<span class="hashtag">#SBATCH</span> --output=outputs/output_%j<br>
<span class="hashtag">#SBATCH</span> --error=errors/error_%j</p>
<p>cd /scratch<br>
mkdir $SLURM_JOB_ID<br>
cd $SLURM_JOB_ID</p>
<p>iget -r /arctic/projects/csc344s73/nsightdemo/arctic_run_folders</p>
<p>source /userapp/virtualenv/SR_ENV/venv/bin/activate<br>
export NCCL_DEBUG=INFO<br>
srun python -u arctic_run_folders/basecode_multinode_multigpu.py</p>
<p>cd /scratch<br>
icd /arctic/projects/csc344s73<br>
iput -rf $SLURM_JOB_ID</p>

shrutishrestha: <p>The nccl one seems working after adding  export NCCL_IB_DISABLE=1</p>

ptrblck: <p>Could you check if adding <code>--ipc=host</code> would also work?</p>

shrutishrestha: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="8" data-topic="156744">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>–ipc=host</p>
</blockquote>
</aside>
<p>where do I keep this?</p>

ptrblck: <p>This flag is used for docker containers. If you are not using any containers, check if <code>ulimit</code> is properly set.</p>