Mismatch vocabulary size between two datasets for NMT task

username: Aiman_Mutasem-bellh
cooked: <p>Hello <span class="mention">@all</span> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a common issue related to continuing to train a pre-training model in machine translation task.</p>
<p>When I load the pre-training model and the new dataset, I got an error related to mismatch between tensor sizes.</p>
<pre><code class="lang-auto">size mismatch for module.decoder.fc.weight: copying a param with shape torch.Size([609, 256]) from checkpoint, the shape in current model is torch.Size([608, 256]).
size mismatch for module.decoder.fc.bias: copying a param with shape torch.Size([609]) from checkpoint, the shape in current model is torch.Size([608]).
</code></pre>
<p>I think this issue related to the vocabulary size <code>self.token_embedding = nn.Embedding(vocab_size, d_model)</code>,  because, whenever I build data <code>build_vocab</code> using only one dataset during pre-training and fine-tuning, everything works fine.</p>
<p>Any suggestions to overcome this issue?</p>
<p>Kind regards,<br>
Aiman Solyman</p>

username: Aiman_Mutasem-bellh
cooked: <pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for MultipleGPUs:
	size mismatch for module.encoder.token_embedding.weight: copying a param with shape torch.Size([875, 256]) from checkpoint, the shape in current model is torch.Size([888, 256]).
	size mismatch for module.decoder.token_embedding.weight: copying a param with shape torch.Size([628, 256]) from checkpoint, the shape in current model is torch.Size([608, 256]).
	size mismatch for module.decoder.fc.weight: copying a param with shape torch.Size([628, 256]) from checkpoint, the shape in current model is torch.Size([608, 256]).
	size mismatch for module.decoder.fc.bias: copying a param with shape torch.Size([628]) from checkpoint, the shape in current model is torch.Size([608]).
</code></pre>

username: ptrblck
cooked: <p>As you’ve already explained, the error is most likely raised due to different vocabularies in both datasets.<br>
If you are planning to use both datasets, you should create a common vocabulary. Besides the different number of words or tokens, you would also have to make sure the common words use the same indices.</p>

username: Aiman_Mutasem-bellh
cooked: <p>Thank you sir for your comment, <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>How to create ‘create a common vocabulary’ is there any tool or I have to create a class for this issue? Now, I build the vocabulary using the dataset that has the biggest number of unique words, which the smallest size, each time that I need to load a dataset.</p>

username: ptrblck
cooked: <p>I would probably stick to your current way of creating the vocabulary for a single dataset, but use both instead.<br>
E.g. assuming you are using the unique target indices to create the mapping, you could extend the target list with both datasets and create a <code>set</code> to remove duplicates.</p>
<p>Note again, that you should also verify that both datasets use the same words for the same indices.</p>

username: Aiman_Mutasem-bellh
cooked: <p>Here is a solution :</p>
<pre><code class="lang-auto">TEXT = Field(...)
imdb_train, imdb_test = IMDB.splits(text_field=TEXT, ...)
snli_train, snli_valid, snli_test = SNLI.splits(text_field=TEXT, ...)
TEXT.build_vocab(imdb_train, snli_train)
</code></pre>