Pytorch DDP replica size mismatch error

karthi0804: <p>I am trying to train GNN using DDP multi-node GPUs. I am using pytorch 1.7 and gloo as backend. I get an error like below on machine 2.</p>
<pre><code class="lang-auto">-- Process 0 terminated with the following error:
Traceback (most recent call last):
......
  File "/home/karthi/venvv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 532, in _ddp_init_helper
    self.gradient_as_bucket_view)
RuntimeError: replicas[0][0] in this process with sizes [64, 5] appears not to match sizes of the same param in process 0.
</code></pre>
<p>But the same code works in single machine multi-gpu DDP. Please help me.This text will be hidden</p>

mrshenli: <p>Hey <a class="mention" href="/u/karthi0804">@karthi0804</a>, could you please share code of a repro?</p>
<p>This error is from <a href="https://github.com/pytorch/pytorch/blob/7a8c64da4d93fef5986aee5ada59289b5387cf8e/torch/lib/c10d/reducer.cpp#L317" rel="noopener nofollow ugc">here</a>, indicating that parameter sizes/order do not match across processes. Could you please verify if that is the case by printing sizes of params in <code>model.parameters()</code>?</p>
<p>cc <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a></p>

karthi0804: <p>My bad. Extremely sorry! silly mistake. Wrong model architecture in another machine!</p>

BlockWaving: <p>Hi, I am also getting this replicas error…<br>
I am using windows 10, torch 1.7.1, pytorch-lightning 1.1.7 with 3 gpus.</p>
<p>The model training was working well with ddp and 2 gpus, on another machine (win10, torch 1.7.1 and pl 1.1.7)</p>
<p>the code crashed after printed the following error message:</p>
<pre><code>self.reducer = dist.Reducer(
</code></pre>
<p>RuntimeError: replicas[0][0] in this process with sizes [12, 6] appears not to match sizes of the same param in process 0.</p>
<p>Please help!</p>

karthi0804: <p>My issue was that I failed to update the model in the second machine as the same version of the first machine. Pls, check it by printing and comparing the model params.</p>

BlockWaving: <p>Thanks check back <a class="mention" href="/u/karthi0804">@karthi0804</a>,  i copied exact model and all supporting code over from previous machine, because this new machine is just built from scratch.   did a lot of other tests, the new machine also failed at same replicas error when tested with two gpus (by physically disconnecting the 3rd one).</p>
<p>however, after tinkering around and setting pl.Trainer with ‘accelerator=ddp_spawn’ instead of ‘ddp’, it works, even though the pytorch lightning docs explicitly warns against using the former.  I don’t exactly know what the limitations they warn about ‘ddp_spawn’–so far i can not observe one with naked eyes, hopefully we can fix the error with ‘ddp’ here or there and switch back.</p>

gqk: <p>Hi, I encountered this issue when using the following 2d group convolution in my code:</p>
<pre><code class="lang-auto">nn.Conv2d(n_feat, n_feat, kernel_size=5, padding=2, groups=n_feat)
</code></pre>
<p>The 2d group convolution worked well in my previous code with DDP mode. After transferring my model to a new framework, it reported the replica mismatch error. I also tested other models in the new framework with the 2d group convolution, and they all worked well.</p>
<p>The environment is based on the docker image ‘pytorch/pytorch:1.7.0-cuda11.0-cudnn8-runtime’.</p>
<p>I really need help!</p>