Torch.distributed.barrier is not working

Saurav_Gupta1: <p>Hi. As per my understanding <code>torch.distributed.barrier</code> will put the first process on hold until all the other processes has reached to the same point. Well it does not seem to be working on the cluster. I am using nccl backend. I have added some print statements just to debug.</p>
<p>My code:</p>
<pre><code class="lang-auto">def init_process(rank, size, backend='nccl'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)

def train(rank, world_size):
  
    print("inside train method")

    init_process(rank, world_size, "nccl")
    print(
        f"Rank {rank + 1}/{world_size} process initialized."
    )
    
    if rank == 0:  
        get_dataloader(rank, world_size)
        model.gru()
    dist.barrier()
    print(f"Rank {rank + 1}/{world_size} training process passed data download barrier.")

def main():
    mp.spawn(train,
        args=(WORLD_SIZE,),
        nprocs=WORLD_SIZE,
        join=True)
</code></pre>
<p>actual output:</p>
<pre><code class="lang-auto">inside train method
Rank 2/2 process initialized
Rank 1/2 training process passed data download barrier
starting epochs
Rank 1/2 process initialized
Rank 0/2 training process passed data download barrier
starting epochs
</code></pre>
<p>I expected output to be like:<br>
<code>inside train method.</code><br>
<code>Rank 1/2 process initialized.</code><br>
<code>Rank 2/2 process initialized.</code><br>
<code>Rank 1/2 training process passed data download barrier.</code><br>
<code>Rank 2/2 training process passed data download barrier.</code><br>
<code>starting epochs</code><br>
<code>starting epochs.</code></p>
<p>From this it is clear that first process did not wait for second process. Am I missing something?</p>

osalpekar: <p>I’m a bit confused by the output logs. How does Rank 1 log “process initialized” after logging “training process passed data download barrier starting epochs”? Also why are there 3 ranks with a world size of 2? Lastly, “inside train method” should be logged twice if you are spawning 2 processes.</p>
<p>I reproduced your script, and swapped out the <code>get_dataloader()</code> and <code>model.gru()</code> calls with a print statement that prints “Just before barrier”. I got the expected output:</p>
<pre><code class="lang-auto">spawning
inside train method
inside train method
Rank 1/2 process initialized.
Rank 2/2 process initialized.
Rank 1/2 Just before barrier
Rank 1/2 training process passed data download barrier.
Rank 2/2 training process passed data download barrier.
</code></pre>
<p>barrier() should indeed block until all the nodes reach the barrier. It would be worth debugging if there is something odd occuring in the <code>get_dataloader()</code> and <code>model.gru()</code> calls. For a lighter weight barrier implementation, you can try calling <code>dist.allreduce</code> on a very small (or empty) torch tensor and this should mimic the barrier behavior.</p>

Saurav_Gupta1: <p>sorry my bad my output is</p>
<pre><code class="lang-auto">inside train method
Rank 2/2 process initialized
Rank 2/2 training process passed data download barrier
starting epochs
Rank 1/2 process initialized
Rank 1/2 training process passed data download barrier
starting epochs
</code></pre>
<p>But I expected output to be like yours.</p>