Why is the 0 vector still not returned after set padding_idx?

Ezra_Corli: <p>When I use nn.Embedding() I set padding_idx is equal to my padding num. But these padding tokens are not represented as the 0 vector.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">self.tok_embed = nn.Embedding(vocab_size, d_model, padding_idx=1)  # token embedding
    self.seg_embed = nn.Embedding(n_segments, d_model, padding_idx=2)  # segment(token type) embedding

    def forward(self, x, seg):
        print('------------x------------')
        print(x)
        print('------------seg------------')
        print(seg)
        embedding = self.tok_embed(x) + self.seg_embed(seg)
        print('------------tok_emb------------')
        print(self.tok_embed(x))
        print('------------seq_emb------------')
        print(self.tok_embed(x))
        print('------------emb------------')
        print(embedding)
        print('------------norm_emb------------')
        print(self.norm(embedding))
        return self.norm(embedding)
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">    self.tok_embed = nn.Embedding(vocab_size, d_model, padding_idx=1)  # token embedding
    self.seg_embed = nn.Embedding(n_segments, d_model, padding_idx=2)  # segment(token type) embedding

    def forward(self, x, seg):
        print('------------x------------')
        print(x)
        print('------------seg------------')
        print(seg)
        embedding = self.tok_embed(x) + self.seg_embed(seg)
        print('------------tok_emb------------')
        print(self.tok_embed(x))
        print('------------seq_emb------------')
        print(self.tok_embed(x))
        print('------------emb------------')
        print(embedding)
        print('------------norm_emb------------')
        print(self.norm(embedding))
        return self.norm(embedding)
The output is:

------------x------------
tensor([[ 0,  8, 16,  ..., 15,  7,  2],
        [ 0, 18, 16,  ...,  7,  2,  1],
        [ 0, 17, 10,  ...,  7,  2,  1],
        ...,
        [ 0, 20, 23,  ...,  7,  2,  1],
        [ 0,  5,  5,  ...,  7,  2,  1],
        [ 0,  5,  4,  ...,  7,  2,  1]], device='cuda:0')
------------seg------------
tensor([[0, 0, 0,  ..., 1, 1, 1],
        [0, 0, 0,  ..., 1, 1, 2],
        [0, 0, 0,  ..., 1, 1, 2],
        ...,
        [0, 0, 0,  ..., 1, 1, 2],
        [0, 0, 0,  ..., 1, 1, 2],
        [0, 0, 0,  ..., 1, 1, 2]], device='cuda:0')
------------tok_emb------------
tensor([[[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0002,  0.0020,  0.0224,  ...,  0.0045, -0.0242,  0.0187],
         [-0.0066, -0.0321, -0.0200,  ..., -0.0027,  0.0259,  0.0017],
         ...,
         [ 0.0064, -0.0281,  0.0082,  ..., -0.0049, -0.0573, -0.0088],
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022]],

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [ 0.0076, -0.0314, -0.0021,  ...,  0.0129, -0.0129, -0.0231],
         [-0.0066, -0.0321, -0.0200,  ..., -0.0027,  0.0259,  0.0017],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]],

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0227,  0.0028, -0.0010,  ...,  0.0075, -0.0017, -0.0003],
         [ 0.0004, -0.0107, -0.0049,  ..., -0.0434,  0.0116, -0.0139],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]],

        ...,

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0093,  0.0166,  0.0342,  ...,  0.0432, -0.0039, -0.0307],
         [-0.0124,  0.0273, -0.0156,  ..., -0.0013, -0.0422,  0.0077],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]],

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0197, -0.0082, -0.0128,  ...,  0.0115, -0.0088, -0.0391],
         [-0.0197, -0.0082, -0.0128,  ...,  0.0115, -0.0088, -0.0391],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]],

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0197, -0.0082, -0.0128,  ...,  0.0115, -0.0088, -0.0391],
         [-0.0040, -0.0118,  0.0104,  ...,  0.0021,  0.0031,  0.0103],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]]],
       device='cuda:0', grad_fn=&lt;EmbeddingBackward0&gt;)
------------seq_emb------------
tensor([[[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0002,  0.0020,  0.0224,  ...,  0.0045, -0.0242,  0.0187],
         [-0.0066, -0.0321, -0.0200,  ..., -0.0027,  0.0259,  0.0017],
         ...,
         [ 0.0064, -0.0281,  0.0082,  ..., -0.0049, -0.0573, -0.0088],
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022]],

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [ 0.0076, -0.0314, -0.0021,  ...,  0.0129, -0.0129, -0.0231],
         [-0.0066, -0.0321, -0.0200,  ..., -0.0027,  0.0259,  0.0017],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]],

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0227,  0.0028, -0.0010,  ...,  0.0075, -0.0017, -0.0003],
         [ 0.0004, -0.0107, -0.0049,  ..., -0.0434,  0.0116, -0.0139],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]],

        ...,

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0093,  0.0166,  0.0342,  ...,  0.0432, -0.0039, -0.0307],
         [-0.0124,  0.0273, -0.0156,  ..., -0.0013, -0.0422,  0.0077],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]],

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0197, -0.0082, -0.0128,  ...,  0.0115, -0.0088, -0.0391],
         [-0.0197, -0.0082, -0.0128,  ...,  0.0115, -0.0088, -0.0391],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]],

        [[ 0.0035, -0.0105, -0.0110,  ..., -0.0011, -0.0039,  0.0102],
         [-0.0197, -0.0082, -0.0128,  ...,  0.0115, -0.0088, -0.0391],
         [-0.0040, -0.0118,  0.0104,  ...,  0.0021,  0.0031,  0.0103],
         ...,
         [-0.0200,  0.0010, -0.0244,  ...,  0.0049,  0.0289,  0.0029],
         [ 0.0194,  0.0021, -0.0093,  ..., -0.0177,  0.0419, -0.0022],
         [-0.0041,  0.0231, -0.0089,  ..., -0.0152,  0.0088,  0.0188]]],
       device='cuda:0', grad_fn=&lt;EmbeddingBackward0&gt;)
</code></pre>
<p>As you can see on the embedding of x the last tokens are 1s, they represent padding, but it’s not an all-0 vector on the embedding, Instead [-0.0041, 0.0231, -0.0089,…, -0.0152, 0.0088, 0.0188], why is this?</p>

ptrblck: <p>I cannot reproduce the issue using this simple code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">vocab_size = 10
d_model = 8
n_segments = 10

device = "cuda"

tok_embed = nn.Embedding(vocab_size, d_model, padding_idx=1).to(device)
seg_embed = nn.Embedding(n_segments, d_model, padding_idx=2).to(device)

x = torch.tensor([0, 1, 2, 3]).to(device)

out1 = tok_embed(x)
out2 = seg_embed(x)

out1
# tensor([[-0.9138, -0.3469, -0.0994, -0.7238,  0.2816,  0.7308, -0.0581,  0.1455],
#         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
#         [-0.8898, -0.0624, -1.0028,  0.5711, -1.7098, -0.6300,  0.9219,  0.7129],
#         [-0.9700, -0.6591, -0.0065,  0.2430,  0.7822, -0.1491, -0.9139,  0.5997]],
#        device='cuda:0', grad_fn=&lt;EmbeddingBackward0&gt;)
out2
# tensor([[ 0.1007, -0.4595,  1.2587,  0.7451, -1.1481,  1.4298, -1.2909,  1.9153],
#         [-0.2414, -1.1177, -1.3125, -0.0434,  0.9630, -1.5259, -0.4135,  1.7336],
#         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
#         [-0.6549,  0.4560,  1.6745, -0.0908,  2.1517, -0.4591,  1.6650,  2.1112]],
#        device='cuda:0', grad_fn=&lt;EmbeddingBackward0&gt;)
</code></pre>
<p>Could you post a minimal and executable code snippet reproducing the issue in your setup?</p>