DDP RuntimeError: replicas[0][64] in this process with sizes [688, 64] appears not to match sizes of the same param in process 0

username: Fhunmie
cooked: <p>I am getting this replicas error today.</p>
<p><strong>Setup:</strong><br>
Machine : HPC cluster, GPU queue, Single node.<br>
pytorch 1.10.2<br>
Number of GPU devices available:  2<br>
Names of GPU devices:  [‘Tesla V100-PCIE-16GB’, ‘Tesla V100-PCIE-16GB’]<br>
The model training was working well with ddp on 1 gpu .<br>
But gives this error with multiple gpu</p>
<pre><code class="lang-auto">RuntimeError: replicas[0][64] in this process with sizes [688, 64] appears not to match sizes of the same param in process 0.
</code></pre>
<p>I tried this:</p>
<pre><code class="lang-auto"> for param in model.parameters():
            print("param ",param.size())
</code></pre>
<p>I realized that the param sizes were equal until the last 4:<br>
The first gpu</p>
<pre><code class="lang-auto">param  torch.Size([688, 64])
param  torch.Size([688])
param  torch.Size([717, 64])
param  torch.Size([688, 64])
</code></pre>
<p>The second gpu</p>
<pre><code class="lang-auto">param  torch.Size([699, 64])
param  torch.Size([699])
param  torch.Size([730, 64])
param  torch.Size([699, 64])
</code></pre>
<p>What can I do to my model to make the model parameters equal on all devices?</p>
<pre><code class="lang-auto"> transformer = Seq2SeqTransformer(num_encoder_layers=2,
                                        num_decoder_layers = 2,
                                        emb_size = 64,
                                        nhead = 2, 
                                        src_vocab_size=SRC_VOCAB_SIZE, 
                                        tgt_vocab_size =TGT_VOCAB_SIZE, 
                                        dim_feedforward=64,
                                        dropout = 0.2)

model = transformer.to(rank)
model = DDP(model,device_ids =[rank],find_unused_parameters=True)
</code></pre>
<p>Thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Fhunmie" data-post="1" data-topic="160622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/c2a13f/48.png" class="avatar"> Fhunmie:</div>
<blockquote>
<p>The model training was working well with ddp on 1 gpu .</p>
</blockquote>
</aside>
<p><code>DDP</code> should fall back to a standard single GPU use case if only one device is detected.</p>
<aside class="quote no-group" data-username="Fhunmie" data-post="1" data-topic="160622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/c2a13f/48.png" class="avatar"> Fhunmie:</div>
<blockquote>
<p>I realized that the param sizes were equal until the last 4:</p>
</blockquote>
</aside>
<p>Could you check where these parameters are defined and speculate why they would change their shape?<br>
Are you initializing some parameters lazily during the forward pass?</p>

username: Fhunmie
cooked: <p>Thanks, I did check through and the following variables are changing their values</p>
<pre><code class="lang-auto">SRC_VOCAB_SIZE = len(vocab_transform[SRC_LANGUAGE])
TGT_VOCAB_SIZE = len(vocab_transform[TGT_LANGUAGE])
</code></pre>
<p>On the first GPU</p>
<pre><code class="lang-auto">SRC_VOCAB_SIZE  = 717
TGT_VOCAB_SIZE  = 688
</code></pre>
<p>On the second GPU</p>
<pre><code class="lang-auto">SRC_VOCAB_SIZE = 730
TGT_VOCAB_SIZE = 699
</code></pre>
<p>This arises due to my script running the number of GPU times, this means all my preprocessing steps and import statements run 1+ number of GPU times.</p>
<p>I tested it with constant values to the variables it works fine. I would like to know why my preprocessing steps are running multiple times. Is there something I am doing wrong?</p>

username: ptrblck
cooked: <p>Each process will run the script you are launching. Assuming your data loading is in the <code>main</code> method, each process is expected to load and process its own <code>Dataset</code>. This is also the reason why a <code>DistributedRandomSampler</code> is used, which avoids sample duplication.<br>
Make sure that each rank sees the full vocab so that the models are initialized in the same way.</p>