Distributed.all_reduce returns strange results

KaiHoo: <p>Hello, I use DDP module to train ImageNet. To collect training metrics from different GPUs, I use distributed.all_reduce. Here are some related codes:</p>
<pre><code class="lang-auto">local_rank = args.local_rank
torch.cuda.set_device(local_rank)
device = torch.device("cuda", local_rank)

for epoch in range(args.num_epoch + args.warmup_epoch):    
    start = time.time()
    train_loss, train_acc = utils.train_one_epoch(net, train_loader, criterion, optimizer, scheduler)
    val_loss, val_acc = utils.test_one_epoch(net, val_loader, criterion)
    #train_loss, train_acc, val_loss, val_acc are floating numbers 
    reduce_tensor = torch.tensor([train_loss, train_acc, val_loss, val_acc]).to(device)
    torch.distributed.all_reduce(reduce_tensor)
    reduce_tensor /= args.num_gpus
    # args.num_gpus = 8
    time_used = (time.time() - start) / 60.

    if local_rank == 0:
        print('Epoch %d train loss %.3f acc: %.3f%%; val loss: %.3f acc %.3f%%; use %.3f mins.'%
            (epoch, reduce_tensor[0], reduce_tensor[1], reduce_tensor[2], reduce_tensor[3], time_used))
</code></pre>
<p>I only get wrong results in the last epoch. Here are some logs:</p>
<p>log1:</p>
<pre><code class="lang-auto">Epoch 97 train loss 0.892 acc: 77.805%; val loss: 0.930 acc 77.010%; use 8.296 mins.
Epoch 98 train loss 0.887 acc: 77.922%; val loss: 0.931 acc 77.024%; use 8.305 mins.
Epoch 99 train loss 0.422 acc: 38.989%; val loss: 0.459 acc 38.506%; use 8.300 mins.
</code></pre>
<p>All metrics are 4/8 of expected. It seems that results from 4 GPUs are 0.</p>
<p>log2:</p>
<pre><code class="lang-auto">Epoch 96 train loss 0.973 acc: 75.933%; val loss: 0.967 acc 76.188%; use 9.449 mins.
Epoch 97 train loss 0.969 acc: 76.003%; val loss: 0.967 acc 76.148%; use 9.459 mins.
Epoch 98 train loss 0.969 acc: 76.029%; val loss: 0.967 acc 76.228%; use 9.445 mins.
Epoch 99 train loss 1.333 acc: 104.523%; val loss: 1.326 acc 104.876%; use 9.452 mins.
</code></pre>
<p>All metrics are 11/8 of expected: 1.333 / (11/8)=0.969. It seems that results from 3 GPUs are repeated in all reduce. <strong>The strange results only happen in the final epoch</strong>. What could be the possible reasons?</p>
<p>Thanks!</p>

iffiX: <p>Your program seems to be correct, some questions:</p>
<ol>
<li>what backend are you using?</li>
</ol>
<p>and could you please add these tests?:</p>
<ol>
<li>change device to “cpu”, will the error be the same?</li>
<li>print <code>rank, train_loss, train_acc, val_loss, val_acc</code> in each of your process, before <code>all_reduce</code>
</li>
</ol>
<p>theoretically this problem should not happen</p>
<ol>
<li>the default gloo backend supports gpu <code>all_reduce</code> and <code>broadcast</code>
</li>
<li>you cannot repeat or let out a process, since <code>all_reduce</code> is blocking</li>
</ol>

iffiX: <p>Well, if you use nccl, then it must be a cuda tensor. please go on <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

KaiHoo: <p>Thanks for responding! I use NCCL as backend:</p>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend="nccl")
</code></pre>
<p>If I change the device to ‘cpu’, there is an error: Tensors must be CUDA and dense<br>
I will try to print them before all_reduce, and see what happens.</p>

mrshenli: <p>Hey <a class="mention" href="/u/kaihoo">@KaiHoo</a> can you print the <code>reduce_tensor</code> before you pass it to <code>all_reduce</code>, so that we can narrow down whether it is the <code>all_reduce</code> or the DDP training/testing that’s mal-bahaving.</p>

KaiHoo: <p><a class="mention" href="/u/iffix">@iffiX</a> <a class="mention" href="/u/mrshenli">@mrshenli</a> Hello, sorry for the late reply. Here are part of the codes:</p>
<pre><code class="lang-auto">Epoch 95 train loss 1.056 acc: 74.176%; val loss: 0.954 acc 75.958%; use 12.457 mins.
Epoch 96 train loss 1.048 acc: 74.339%; val loss: 0.949 acc 75.998%; use 12.459 mins.
Epoch 97 train loss 1.028 acc: 74.815%; val loss: 0.946 acc 76.232%; use 12.455 mins.
Epoch 98 train loss 1.027 acc: 74.855%; val loss: 0.946 acc 76.236%; use 12.475 mins.
Rank 5 train loss 1.026 acc: 74.890%; val loss: 0.972 acc 75.600%
Rank 6 train loss 1.025 acc: 74.815%; val loss: 0.924 acc 76.352%
Rank 3 train loss 1.025 acc: 74.889%; val loss: 0.957 acc 75.632%
Rank 1 train loss 1.032 acc: 74.757%; val loss: 0.929 acc 76.960%
Rank 7 train loss 1.023 acc: 75.038%; val loss: 0.930 acc 76.512%
Rank 2 train loss 1.019 acc: 75.013%; val loss: 0.958 acc 76.144%
Exception ignored in: &lt;function WeakValueDictionary.__init__.&lt;locals&gt;.remove at 0x7ffb5070eea0&gt;
Traceback (most recent call last):
  File "/usr/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Rank 0 train loss 1.022 acc: 74.841%; val loss: 0.951 acc 76.304%
Epoch 99 train loss 0.513 acc: 37.451%; val loss: 0.470 acc 38.097%; use 12.467 mins.
Rank 4 train loss 1.023 acc: 74.984%; val loss: 0.947 acc 76.304%
</code></pre>
<p>Since the strange results only happen in the final epoch, I only print the metrics for the last epoch. The order of logs are exactly what I got, though the ‘Rank 4’ line should be printed before the ‘Epoch 99 train’ line:</p>
<pre><code class="lang-auto">for epoch in range(args.num_epoch + args.warmup_epoch):    
    start = time.time()
    train_loss, train_acc = utils.train_one_epoch(net, train_loader, 
        criterion, optimizer, mean_and_std, scheduler, args)
    val_loss, val_acc = utils.test_one_epoch(net, val_loader, criterion, mean_and_std)
    reduce_tensor = torch.tensor([train_loss, train_acc, val_loss, val_acc]).to(device)
    if epoch == args.num_epoch + args.warmup_epoch - 1:
        print('Rank %d train loss %.3f acc: %.3f%%; val loss: %.3f acc %.3f%%'%
            (local_rank, reduce_tensor[0], reduce_tensor[1], reduce_tensor[2], reduce_tensor[3]))
    torch.distributed.all_reduce(reduce_tensor)
    reduce_tensor /= args.num_gpus
    time_used = (time.time() - start) / 60.

    if local_rank == 0:
        print('Epoch %d train loss %.3f acc: %.3f%%; val loss: %.3f acc %.3f%%; use %.3f mins.'%
            (epoch, reduce_tensor[0], reduce_tensor[1], reduce_tensor[2], reduce_tensor[3], time_used))
</code></pre>

mrshenli: <aside class="quote no-group" data-username="KaiHoo" data-post="7" data-topic="89248">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaihoo/48/22284_2.png" class="avatar"> KaiHoo:</div>
<blockquote>
<pre><code class="lang-auto">Exception ignored in: &lt;function WeakValueDictionary.__init__.&lt;locals&gt;.remove at 0x7ffb5070eea0&gt;
Traceback (most recent call last):
  File "/usr/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
</code></pre>
</blockquote>
</aside>
<p>Is the above error expected? How did you handle this? If this is handled by skipping/redoing that iteration, it might cause allreduce mismatch.</p>

KaiHoo: <aside class="quote no-group" data-username="KaiHoo" data-post="7" data-topic="89248">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaihoo/48/22284_2.png" class="avatar"> KaiHoo:</div>
<blockquote>
<p><code>File "/usr/lib/python3.5/weakref.py", line 117, in remove</code></p>
</blockquote>
</aside>
<p>I have no idea about this error, though nothing happened. This bug is reported to PyTorch, but seems a bug of python:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/2229">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/2229" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/2229" target="_blank" rel="noopener nofollow ugc">[minor] Random ignored exception on exit of pytorch scripts</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-07-28" data-time="11:37:13" data-timezone="UTC">11:37AM - 28 Jul 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2017-08-03" data-time="00:14:15" data-timezone="UTC">12:14AM - 03 Aug 17 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/cdluminate" target="_blank" rel="noopener nofollow ugc">
          <img alt="cdluminate" src="https://avatars.githubusercontent.com/u/5723047?v=4" class="onebox-avatar-inline" width="20" height="20">
          cdluminate
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">This happends randomly and I have no idea under what condition can this problem <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">be reproduced.

```
Exception ignored in: &lt;function WeakValueDictionary.__init__.&lt;locals&gt;.remove at 0x7f34966fc9d8&gt;
Traceback (most recent call last):
  File "/usr/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
```</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/23760">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/23760" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/23760" target="_blank" rel="noopener nofollow ugc"> torch.nn.parallel.data_parallel crashes machine and has a weakref bug</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-08-03" data-time="17:04:10" data-timezone="UTC">05:04PM - 03 Aug 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2022-02-07" data-time="19:46:22" data-timezone="UTC">07:46PM - 07 Feb 22 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/PetrochukM" target="_blank" rel="noopener nofollow ugc">
          <img alt="PetrochukM" src="https://avatars.githubusercontent.com/u/7424737?v=4" class="onebox-avatar-inline" width="20" height="20">
          PetrochukM
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          high priority
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: data parallel
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🐛 Bug

For some reason, this happened:

```
Exception ignored in: &lt;funct<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ion WeakValueDictionary.__init__.&lt;locals&gt;.remove at 0x7f4b13e53158&gt;
Traceback (most recent call last):
  File "/usr/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
```

## To Reproduce

```
import torch


class Model(torch.nn.Module):

    def forward(self, a):
        return a


device = torch.device('cuda')
model = Model()
model.to(device)
a = torch.randint(0, 255, (256, 900, 2), device=device)
print('`torch.nn.parallel.data_parallel`')
torch.nn.parallel.data_parallel(module=model, inputs=(a,))
```

## Environment

Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Debian GNU/Linux 9.9 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: Could not collect

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 10.1.105
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB

Nvidia driver version: 418.67
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] intel-numpy==1.15.1
[pip3] numpy==1.15.1
[pip3] torch==1.1.0
[conda] Could not collect</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Benjamin_Lefaudeux: <p>I’m a bit curious about this bug, did anyone got to the bottom of this ? <a class="mention" href="/u/kaihoo">@KaiHoo</a>, just in case, are you spreading the GPUs across different nodes ?</p>

KaiHoo: <p>It is a single node DDP</p>