distributed.all_gather_object() produces multiple additional processes

Taejune: <p>Hi, I’m currently studying pytorch DDP with 8 gpus.</p>
<p>I’m trying to train &amp; validate the model with multi-gpus, and the training seems to work fine.</p>
<p>But in the validation phase, I tried to gather the validation output into rank 0 and print the validation accuracy and loss.<br>
It worked, but when <code>dist.all_gather_object</code> is activated, I find that 7 processes are created additionally.</p>
<p>I think there are some inefficiency in my code.</p>
<p>So my questions would be:</p>
<ol>
<li>Why so much processes are created when I tried to gather values from each gpus?</li>
<li>How to gather results from each gpu to rank 0 properly?</li>
<li>Am I using DDP appropriatly &amp; efficiently?</li>
</ol>
<p>These are my code for each epoch and GPU status:</p>
<pre><code class="lang-auto">    for epoch in range(args.epochs):
        # we have to tell DistributedSampler which epoch this is
        # and guarantees a different shuffling order
        train_loader.sampler.set_epoch(epoch)
        train_loss, train_acc = train(model, train_loader, criterion, optimizer, rank, args)
        
        val_acc, val_loss = valid(model, val_loader, criterion, rank, args)
        
        ## gather
        g_acc, g_loss = torch.randn(world_size), torch.randn(world_size)
        dist.all_gather_object(g_acc, val_acc)
        dist.all_gather_object(g_loss, val_loss)
        
        if rank == 0:
            val_acc, val_loss = g_acc.mean(), g_loss.mean()
            print(f"EPOCH {epoch} VALID: acc = {val_acc}, loss = {val_loss}")
            if val_acc &gt; best_acc:
                save_ckpt({
                    "epoch": epoch+1,
                    "state_dict": model.module.state_dict(),
                    "optimizer": optimizer.state_dict(),
                    "scheduler": scheduler.state_dict(),
                }, file_name=os.path.join(args.exp, f"best_acc.pth"))
            if val_loss &lt; best_loss:
                save_ckpt({
                    "epoch": epoch+1,
                    "state_dict": model.module.state_dict(),
                    "optimizer": optimizer.state_dict(),
                    "scheduler": scheduler.state_dict(),
                }, file_name=os.path.join(args.exp, f"best_loss.pth"))
            save_ckpt({
                    "epoch": epoch+1,
                    "state_dict": model.module.state_dict(),
                    "optimizer": optimizer.state_dict(),
                    "scheduler": scheduler.state_dict(),
                }, file_name=os.path.join(args.exp, f"last.pth"))
        scheduler.step()
        dist.barrier()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d74283892d3a489ae5b2a0c9e103e664ea4c98e0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d74283892d3a489ae5b2a0c9e103e664ea4c98e0" title="스크린샷 2022-11-02 오후 2.01.18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d74283892d3a489ae5b2a0c9e103e664ea4c98e0_2_256x500.jpeg" alt="스크린샷 2022-11-02 오후 2.01.18" data-base62-sha1="uIha09LIv6IY1fbL7cNPVn9I3nO" width="256" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d74283892d3a489ae5b2a0c9e103e664ea4c98e0_2_256x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d74283892d3a489ae5b2a0c9e103e664ea4c98e0_2_384x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d74283892d3a489ae5b2a0c9e103e664ea4c98e0_2_512x1000.jpeg 2x" data-dominant-color="2B2B2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2022-11-02 오후 2.01.18</span><span class="informations">1152×2246 322 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for read!</p>

wanchaol: <p><a class="mention" href="/u/taejune">@Taejune</a> how do you initialize the process_group? all_gather_object itself won’t spawn new processes, it’s mostly user need to create multiple processes with the launcher. could you please check why there’re multiple new processes?</p>

Taejune: <p>Hi, thanks for your reply and sorry for my late reply.<br>
I called <code>dist.init_process_group("nccl", "env://", rank=rank, world_size=8)</code>.<br>
I used the world size of 8 instead of 1 in the code below.<br>
And I used <code>mp.spawn</code> method to create multiple processes to achieve DDP.</p>
<p>Would <code>mp.spawn</code> be a reason for creating such additional processes?</p>
<p>Thanks.</p>
<pre><code class="lang-auto">def setup(rank, world_size, args):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    
    dist.init_process_group("nccl", init_method=args.dist_url,rank=rank, world_size=world_size)
    
def parse_args():
    parser = argparse.ArgumentParser(description="Imagenet Training")
    
    ## Config
    parser.add_argument("--exp", type=str, default="./exp/default")
    
    ## DDP
    parser.add_argument("--dist_url", type=str, default="env://")
    
    ## training
    parser.add_argument("--data_path", type=str, default="/home/data/imagenet")
    parser.add_argument("--lr", type=float, default=1e-4)
    parser.add_argument("--epochs", type=int, default=100)
    parser.add_argument("--batch-size", type=int, default=64)
    parser.add_argument("--valid-iter", type=int, default=1000)
    
    ## data loader
    parser.add_argument("--pin-memory", action='store_true')
    parser.add_argument("--num-workers", type=int, default=2) # may cause a bottleneck if set to be 0
    parser.add_argument("--drop-last", action="store_true")
    parser.add_argument("--shuffle", action="store_true")
    
    return parser.parse_args()

if __name__ == "__main__":
    world_size = 1
    args = parse_args()

    os.makedirs(os.path.join("./exp", args.exp), exist_ok=True)
    
    mp.spawn(
        main,
        args=(world_size, args),
        nprocs=world_size
    )
</code></pre>

Jonathan_Schmidt: <p>Did your issue ever get solved? I have the same issue with a dist.gather call.</p>

wanchaol: <p>I think it would at most create 8 processes and it’s controlled by the <code>mp.spawn</code>. I don’t think we spawn additional processes anywhere else. Maybe the dataloader is having additional processes?</p>