Problem in padding_idx of torch.nn.functional.embedding

username: zqu1992
cooked: <p>There may exist a problem of torch.nn.functional.embedding</p>
<blockquote>
<p>print(torch.nn.functional.embedding(torch.LongTensor([[0,2,0,5]]), torch.rand(10, 3), padding_idx=0))</p>
</blockquote>
<p>The output embedding vectors are not zero if the corresponding index=0.</p>

username: RoySadaka
cooked: <p><a class="mention" href="/u/zqu1992">@zqu1992</a></p>
<p>I can confirm that, and this also happens with the nn.embedding version:</p>
<p><code>predefined_weights = torch.rand(10, 3)</code></p>
<p><code>emb = torch.nn.Embedding(num_embeddings=10, embedding_dim=3, padding_idx=0, _weight=predefined_weights)</code></p>
<p><code>emb(torch.LongTensor([[0,2,0,5]]))</code></p>
<p>It “pads” with the vector at idx=0 from predefined_weights instead of a zero vector.<br>
From the code, it looks like it zeros-out the vector in padding_idx in <code>reset_parameters</code> but only<br>
<code>if _weight is None</code><br>
otherwise, it uses the given _weight parameter.</p>
<p>IMO, the behavior is fine and the documentation should adjust, cause if you provide your own weights, you should get full control over the weights, including the padding vector.</p>
<p>Roy.</p>

username: zqu1992
cooked: <p>Thanks a lot.<br>
I agree the predefined weight can better control the behavior.<br>
But the relevant definition of the argument _weight should be added in the official document anyway.</p>

username: ptrblck
cooked: <p>Would you mind creating an issue on GitHub with this suggestion?<br>
Also, would you or <a class="mention" href="/u/roysadaka">@RoySadaka</a> be interested in providing the PR for it? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: RoySadaka
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Sure ill do it, thank you!</p>

username: RoySadaka
cooked: <p>opened:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/46585" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/46585" target="_blank" rel="noopener nofollow ugc">padding_idx and provided weights in nn.Embedding and nn.functional.embedding </a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-10-20" data-time="11:54:14" data-timezone="UTC">11:54AM - 20 Oct 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/RoySadaka" target="_blank" rel="noopener nofollow ugc">
          <img alt="RoySadaka" src="https://avatars2.githubusercontent.com/u/9369514?v=4" class="onebox-avatar-inline" width="20" height="20">
          RoySadaka
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">1) nn.Embedding documentation says:
padding_idx (int, optional): If given, pads the output with the embedding vector at :attr:padding_idx
(initialized to zeros) whenever it...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>