Padding zeros in nn.Embedding while using pre-train word vectors

HT_Liu: <p>Hi, I have come across a problem in Pytorch about embedding in NLP.<br>
Suppose I have |N| sentences with different  length, and I set the <code>max_len</code> is the max length among the sentences, while the other sentences need to pad zeros vectors. Define the <code>Embedding</code> as below with one extra zero vectors at index <code>vocab_size</code></p>
<pre><code class="lang-auto">emb = nn.Embedding(vocab_size +1, emb_num, padding_idx=vocab_size)
</code></pre>
<p>But when I use my pre-train word vectors: I <strong>need to construct</strong> a vectors including a zero vectors <strong>manually</strong>:</p>
<pre><code class="lang-auto">myw2v=torch.from_numpy('myw2v.model')
zeros = torch.zeros(1, emb_num)
myw2v = torch.cat((myw2v, zeros), 1)
emb.weight.data.copy_(torch.from_numpy(myw2v))
</code></pre>
<p>So  I’d like to know whether there is <strong>a function in pytorch</strong> that can handle the situation above?<br>
Thanks.</p>

Siddharth_Vashishtha: <p>Did you figure this out? I’d also like to know the same.</p>

HT_Liu: <p>I just manually using <code>torch.cat</code> to do this job and can not find out the function in Pytorch</p>

o2h4n: <p>As far as I understand (vocab_size +1) is the maximum unique element for embedding. Your input size does not have to be the vocab_size.<br>
When you use padding_idx=vocab_size you are saying that input will always have the size of your vocabulary and this is not the case.<br>
There can be 2 solutions:</p>
<ol>
<li>You can give padding_idx the size of your input after padding <code>( max_len + 1)</code>
</li>
<li>You can add zero at the beginning of each sentence with <code>padding_idx=0</code>. After embedding, remove the first embedded element and give it as input to the next layer.</li>
</ol>
<p>I hope it is clear.</p>
<p>This is an example (I create my input by adding zero(my padding) at the beginning ) but you can use cat like this:</p>
<pre><code class="lang-auto">input = torch.tensor([[8, 1, 2, 5],[9, 6, 0, 0]])
zeros = torch.zeros(2, 1).long()
input = torch.cat((zeros,input), 1)
print(input)
emb = nn.Embedding(10,3,padding_idx=0)
embedded = emb(input)[:,1:]
print(embedded)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor([[0, 8, 1, 2, 5],
        [0, 9, 6, 0, 0]])
tensor([[[ 1.5745, -2.3024, -0.5964],
         [-1.5407,  0.8915, -1.3858],
         [-1.2728,  1.5402,  0.7473],
         [ 0.1958,  0.4828,  0.5091]],

        [[-0.1123, -0.3911,  0.4758],
         [ 0.0876, -0.6896,  0.0802],
         [ 0.0000,  0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000]]], grad_fn=&lt;SliceBackward&gt;)
</code></pre>

vamshi123: <p><a class="mention" href="/u/smth">@smth</a> could you please help us in this issue!!! Thanks !</p>

Keren_Gaiger: <p>You can specify a certain index to represent the ‘zero padding’ with torch.nn.Embedding(padding_idx=,…). Then, for each sentence of size less than max_len, create a list in the size of (max_len-sentence size) , of which its elements are actually the paddind_idx index. Then add this list to the already existing list of word indices representing the sentence.</p>

Etienne_Perot: <p>In my case the padding index was -1.<br>
I did:<br>
<code>emb = nn.Embedding(vocab_size +1, emb_num, padding_idx=vocab_size)</code><br>
my input sequence is named ‘tgt’</p>
<pre><code class="lang-auto">tgt_pad = tgt.clone()
tgt_pad[tgt == -1] = self.vocab_size
emb_vec = emb(tgt_pad)
</code></pre>