How to validate in DistributedDataParallel correctly?

username: RocketFlash
cooked: <p>I am trying to train and validate model using DistributedDataParallel. Everything is fine during training, but when the model starts validate, the code works several iterations and after crashes due to errors with threads. I do validation only in rank=0. Do I need to put dist.barrier() somewhere? Or do I need to validate in all ranks?</p>

username: mrshenli
cooked: <p>When rank0 validates the model, what do other ranks do? Exit or proceed to the next training phase? And what error did you see?</p>
<p>I am assuming other ranks proceed to the next training phase and then timeout during DDP backward pass. If this is the case, yep, you can try use a <code>dist.barrier()</code> to sync, like:</p>
<pre><code class="lang-python">for _ in range(...):
    train()
    if rank == 0:
        validate()
    dist.barrier()
</code></pre>
<p>If you still hit timeout at barrier, you can try setting the timeout arg in <code>init_process_group</code> to a larger value.</p>

username: RocketFlash
cooked: <p>It seems I solved the problem. I used DistributedSampler in validation dataloader, now I changed sampler to None and now code works. Other ranks do nothing. I don’t know how to share validation loss values across ranks, so I do validation only in rank=0. Is it a good practice to do so? Also I am using wandb to monitor training metrics, but after several epochs I got Timeout error at barrier. Is it correct to increase the value of  timeout limit in this situation?</p>

username: ayalaa2
cooked: <p>Take a look <a href="https://pytorch.org/docs/stable/distributed.html#collective-functions" rel="nofollow noopener">here</a> on how to share information between your ranks.</p>
<p>Since you’re using wandb, I’m assuming that you’re also only logging with rank 0 as well? I wouldn’t say it’s bad practice to share the validation loss with all your ranks if the other ranks aren’t doing anything with the information. It’s important when you’re doing early stopping or learning rate scheduling based off of the validation loss.</p>
<p>Personally, I had all my ranks participate in computing the validation loss. In this scenario, you wouldn’t have to deal with barriers and instead just share the tensors containing your losses. But I don’t think it’s necessarily wrong to increase the timeout limit.</p>

username: RocketFlash
cooked: <p>Thank you, <a class="mention" href="/u/ayalaa2">@ayalaa2</a> ! I understand that if I used all the ranks for validation it would be faster. I will try to share tensor values across ranks.</p>
<aside class="quote no-group" data-username="ayalaa2" data-post="4" data-topic="94267">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ayalaa2/48/21195_2.png" class="avatar"> ayalaa2:</div>
<blockquote>
<p>ssuming that you’re also only logging with rank 0 as well?</p>
</blockquote>
</aside>
<p>Yes, I am using wandb only with rank 0.</p>

username: RocketFlash
cooked: <p><a class="mention" href="/u/ayalaa2">@ayalaa2</a> could you give some example code how to use sharing data between ranks, please?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="RocketFlash" data-post="6" data-topic="94267">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rocketflash/48/28255_2.png" class="avatar"> RocketFlash:</div>
<blockquote>
<p>could you give some example code how to use sharing data between ranks, please?</p>
</blockquote>
</aside>
<p>One option is to use the collective communication APIs, e.g., <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather" rel="noopener nofollow ugc"><code>all_gather</code></a> (NCCL and Gloo) or <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.gather" rel="noopener nofollow ugc"><code>gather</code></a> (Gloo). Sth like:</p>
<pre data-code-wrap="python"><code class="lang-python"># on all ranks

out_tensors = [torch.zeros(2, 2), torch.zeros(2, 2)]
inp_tensor = [torch.ones(2, 2)]

torch.distributed.all_gather(out_tensors, inp_tensor)

</code></pre>

username: RocketFlash
cooked: <p>Thank you very much, <a class="mention" href="/u/mrshenli">@mrshenli</a>!</p>
<p>Is it possible to store not only tensors, but also list of strings for example?</p>

username: mrshenli
cooked: <blockquote>
<p>Is it possible to store not only tensors, but also list of strings for example?</p>
</blockquote>
<p>Yep. But the feature is only available on master (will be released in v1.7). See the following code:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L580-L600">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L580-L600" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L580-L600" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L580-L600</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="580" style="counter-reset: li-counter 579 ;">
          <li>def test_event_multiprocess(self):</li>
          <li>    event = torch.cuda.Event(enable_timing=False, interprocess=True)</li>
          <li>    self.assertTrue(event.query())</li>
          <li></li>
          <li>    ctx = mp.get_context('spawn')</li>
          <li>    p2c = ctx.SimpleQueue()</li>
          <li>    c2p = ctx.SimpleQueue()</li>
          <li>    p = ctx.Process(</li>
          <li>        target=TestMultiprocessing._test_event_multiprocess_child,</li>
          <li>        args=(event, p2c, c2p))</li>
          <li>    p.start()</li>
          <li></li>
          <li>    c2p.get()  # wait for until child process is ready</li>
          <li>    torch.cuda._sleep(50000000)  # spin for about 50 ms</li>
          <li>    event.record()</li>
          <li>    p2c.put(0)  # notify child event is recorded</li>
          <li></li>
          <li>    self.assertFalse(event.query())</li>
          <li>    c2p.get()  # wait for synchronization in child</li>
          <li>    self.assertTrue(event.query())</li>
          <li>    p.join()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If you need it now, you can copy the code change in <code>distributed_c10d.py</code> from <a href="https://github.com/pytorch/pytorch/pull/42189/files" rel="noopener nofollow ugc">this PR</a>. It basically is a wrapper that converts strings to tensors.</p>

username: dang-qi
cooked: <p>Hi, I used similar code like this:</p>
<pre><code>    for i in range( self._epoch+1, self._niter+1 ):
        if self._train_sampler is not None:
            self._train_sampler.set_epoch(i)
        self._train()           
        if self.rank==0:
            print('rank {} go to validation'.format(self.rank))
            self._validate()
        if self.distributed:
            print('rank {} go to barrier'.format(self.rank))
            dist.barrier()
            print('rank {} go out of barrier'.format(self.rank))
</code></pre>
<p>The output of the code is like:<br>
rank 1 go to barrier<br>
Training…<br>
rank 0 go to validation<br>
start to validate<br>
evaluating…<br>
rank 0 go to barrier<br>
rank 0 go out of barrier</p>
<p>Then it just stopped and rank 1 won’t go out. There is no other error, just freezed. Do you have any idea what happened? When I remove the validation, the code is working. My test dataloader didn’t use the distributedSampler.</p>

username: seungjun
cooked: <p>Evaluating with <code>DistributedDataParallel</code> should be done with care otherwise the values could be inaccurate. <code>DistributedSampler</code> can pad some replicated data when the number of samples per process is not even. How <code>DistributedSampler</code> works is explained <a href="https://discuss.pytorch.org/t/distributedsampler/90205/2">here</a>.</p>
<p>This is because DDP checks synchronization at backprops and the number of minibatch should be the same for all the processes. However, at evaluation time it is not necessary.</p>
<p>You can use a custom sampler like <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/data/sampler.py" rel="noopener nofollow ugc">DistributedEvalSampler</a> to avoid data padding. Regarding the communication between the DDP processes, you can refer to <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/loss/__init__.py#L257" rel="noopener nofollow ugc">this example</a>.</p>

username: Euruson
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> The <code>dist.barrier()</code> doesn’t work while running validation only on rank 0, as mentioned in this <a href="https://discuss.pytorch.org/t/distributeddataparallel-barrier-doesnt-work-as-expected-during-evaluation/99867">thread</a>.</p>

username: shaibagon
cooked: <p>Can you elaborate on how you validate using all ranks?</p>

username: ayalaa2
cooked: <p>Do you mean how you would go about performing a validation pass using all ranks?</p>
<p>Your validation loop will operate very similar to your training loop where each rank will operate on a subset of the validation dataset. The only difference is that you will want to keep track of the validation metrics you care about and then share the results using the methods in the link I gave.</p>
<p>Sometimes its quicker to just let the main rank perform validation depending on the specific task.</p>

username: shaibagon
cooked: <p><a class="mention" href="/u/ayalaa2">@ayalaa2</a> thanks for the prompt reply!</p>
<p>Does it mean I need a distributed sampler on the validation set as well?</p>
<p>So, if I want to measure accuracy on my validation, each rank should count the number of “correct” predictions and the number of samples it saw and the “reduce” it?</p>
<p>Would that look something like this:</p>
<pre><code class="lang-auto">def validate(val_loader, model):
  # one counter for "correct" the other for "total"
  counter = torch.zeros((2,), device=torch.device(f'cuda:{args.rank}'))
  model.eval()
  with torch.no_grad():
    for x, y in val_loader:  # assuming val_loader has a distributed sampler
      pred = model(x)
      num_correct = my_accuracy_function(pred, y)  # count number of correct predictions
      counter[0] += num_correct
      counter[1] += x.shape[0]  # total number of samples 
  # done validating for this view
  torch.distributed.reduce(counter, 0)  # reduce all to rank 0 process
  if args.rank == 0:
   # only this rank reports accuracy
   print(f'total accuracy = {counter[0]}/{counter[1]} = {counter[0]/counter[1]}')
</code></pre>
<p>Does that make any sense?</p>

username: ayalaa2
cooked: <p>Yes you would need a distributed sampler to make sure each rank gets a unique data split. The code you posted looks about right.</p>

username: shaibagon
cooked: <p>Thank you! This really helps</p>

username: Yanting_Huang
cooked: <p>I met the same issue</p>