DDP: replicas[0][0] in this process with sizes [12, 6] appears not to match sizes

BlockWaving: <p>Hi,</p>
<p>I am getting this replicas error today.<br>
setup:  windows 10, torch 1.7.1, pytorch-lightning 1.1.7 with 3 gpus.</p>
<p>The model training was working well with ddp and 2 gpus, on another machine (same setup w/  win10, torch 1.7.1 and pl 1.1.7)</p>
<p>the code crashed after printed the following error message:</p>
<pre><code class="lang-auto">self.reducer = dist.Reducer(
RuntimeError: replicas[0][0] in this process with sizes [12, 6] appears not to match sizes of the same param in process 0.
</code></pre>
<p>Please help!</p>

osalpekar: <p>This happens if the model parameters are not the same across all replicas in DDP. Have you tried printing the sizes of all the params in the model from each rank (using <code>model.parameters()</code>)? This would be the first thing to verify mismatched sizes.</p>
<p>Can you also provide your code to repro?</p>

BlockWaving: <p>Hi thanks for quick reply!<br>
I am using pytorch-lightning 1.1.7 on top of the torch 1.7.1.  I don’t directly call torch api’s, but using lightning’s Trainer, and model.fit .</p>
<p>The lightning underline indeed printed out the model parameters, all three are the same,  (they all rounded to thousand though).</p>
<p>The weird thing is that, the training worked very well on first machine, which has two gpus.  The problem happened to the second machine, which has 3 gpus.  But even after i removed 1 gpu from the the machine, training with 2 gpus on this machine still fails w/ replicas error.</p>

BlockWaving: <p>This is how lightning Trainer is initialed and then fit is called:<br>
“”"<br>
self.trainer = pl.Trainer(<br>
max_epochs=configs[“max_epochs”],<br>
gpus=[0, 1],<br>
accelerator=‘ddp’,<br>
weights_summary=“top”,<br>
gradient_clip_val=0.1,<br>
limit_train_batches=30,<br>
callbacks=[lr_logger, early_stop_callback, checkpoint_callback],<br>
)</p>
<p>model = …</p>
<p>self.trainer.fit(<br>
model,<br>
train_dataloader=self.train_dataloader,<br>
val_dataloaders=self.val_dataloader,<br>
)<br>
“”"</p>

BlockWaving: <p>Are there any findings for this?<br>
Later tried with accelerator=‘ddp_spawn’, and the replicas error seemingly disappeared.<br>
But the training with ‘ddp_spawn’ very easily get stuck or crash after a few epochs, with error messages like this:</p>
<blockquote>
<p>File “D:\installed\anaconda3\envs\TorchB\lib\site-packages\pytorch_lightning\trainer\training_loop.py”, line 720, in train_step_and_backward_closure<br>
result = self.training_step_and_backward(<br>
File “D:\installed\anaconda3\envs\TorchB\lib\site-packages\pytorch_lightning\trainer\training_loop.py”, line 828, in training_step_and_backward<br>
self.backward(result, optimizer, opt_idx)<br>
File “D:\installed\anaconda3\envs\TorchB\lib\site-packages\pytorch_lightning\trainer\training_loop.py”, line 850, in backward<br>
result.closure_loss = self.trainer.accelerator_backend.backward(<br>
File “D:\installed\anaconda3\envs\TorchB\lib\site-packages\pytorch_lightning\accelerators\accelerator.py”, line 104, in backward<br>
model.backward(closure_loss, optimizer, opt_idx, *args, **kwargs)<br>
File “D:\installed\anaconda3\envs\TorchB\lib\site-packages\pytorch_lightning\core\lightning.py”, line 1158, in backward<br>
loss.backward(*args, **kwargs)<br>
File “D:\installed\anaconda3\envs\TorchB\lib\site-packages\torch\tensor.py”, line 221, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “D:\installed\anaconda3\envs\TorchB\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 130, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: bad allocation</p>
</blockquote>
<p>Additional info:   I am using ‘gloo’ backend, and init_method=“file:/// …”.</p>

huahuanZ: <p>PyTorch DDP requires the params in all GPUs follow the same order. e.g.<br>
GPU0: weight [4,4], bias [4]<br>
GPU1: bias[4], weight [4,4]<br>
This is invalid.<br>
Try print all the parameters names and theirs sizes to text files. Just make sure they follow the same order.</p>
<pre><code class="lang-auto">with open(f"params_{args.rank}.txt", "w") as fo:
    for name, param in model.parameters():
        fo.write(f"{name}\t{param.size()}\n")
</code></pre>