[nn.EmbeddingBag]: How to weight embeddings before summation

username: weichium
cooked: <p>Hi,</p>
<p>According to the current implementation: for bags of constant length, <code>nn.EmbeddingBag</code> with <code>mode=sum</code> is equivalent<br>
to  <code>nn.Embedding</code> followed by <code>torch.sum(dim=1)</code>. In other words, the embeddings all have the same weight, i.e. <code>1</code>. I wonder if it is possible to weight the embeddings before summing them up or if there is any efficient way to do so?</p>
<p>Currently what I did is first use <code>nn.Embedding</code> to extract embeddings, then multiply with weights, and finally summing them up. This, however, is very inefficient since we need to instantiate the intermediate embeddings. Is it possible to add such feature? Or perhaps providing some hints on how to do this effectively.</p>
<p>Thanks for the help!</p>

username: tom
cooked: <p>Hi,</p>
<p>you can reformulate this as matrix (of stacked embedding vector) multiplied with a (weight-) vector. Then (possibly after sprinkling transpose on the weight vector and result) you can use <a href="http://pytorch.org/docs/master/torch.html#torch.mv" rel="nofollow noopener"><code>torch.mv</code></a>.<br>
For batch operation it might be easier to make the weight vector into a <code>nx1</code> matrix and use <a href="http://pytorch.org/docs/master/torch.html#torch.matmul" rel="nofollow noopener"><code>torch.matmul</code></a>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: weichium
cooked: <p>Hi Thomas,</p>
<p>Thanks for your reply. I guess my question is not clear enough. Here is a more concrete example:</p>
<p>Say I have two sentences of different length. When computing the embedding one the two sentence, one naive way to do this is simply using <code>nn.EmbeddingBag</code> with <code>mode='sum'</code> or <code>mode='mean'</code>. However, what I want is sort of attention on the embeddings, i.e. rather than w_1 + w_2 + … + w_n, I’m looking for a_1 * w_1 + … + a_n * w_n, where both the attention weight <code>a</code> and the embeddings <code>w</code> are learnable. So what I did is first use <code>nn.Embedding</code> to extract the vectors, then weight it. Finally, since multiple sentences may have different length, I used <code>nn.EmbeddingBag</code> to sum the corresponding word embeddings by providing the <code>offsets</code>.</p>
<p>The solution that you suggest seems to me like a <code>dense</code> version, where there will be lots of <code>0</code> in the matrix. I believe it will consumes way more memory. What I’m looking for is a memory efficient (sparse) way to deal with this. Please correct me if I’m misunderstand something <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

username: tom
cooked: <p>Hi,</p>
<p>indeed, you have me confused.<br>
I thought that you had Embeddings “per seen word” from <code>nn.Embedding</code> (and can afford the memory from it) and wanted a weighted sum over them without an intermediate “multiply with weights” step.</p>
<p>As far as I understand, EmbeddingBag avoids the “per seen word” memory allocation by (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/_functions/thnn/sparse.py#L203" rel="nofollow noopener">in its slow cpu version</a>) using <code>Tensor.index_add_</code>. Indeed I am unaware of a way to do this and have the weights applied in the same step.</p>
<p>Best regards</p>
<p>Thomas</p>

username: MANSUM
cooked: <p>Can anyone answer this question?</p>

username: 368e621b293cf07f16ca
cooked: <p>It would also be awesome to train those weights in the weighted sum</p>

username: marcwww
cooked: <p>U could weight the embeddings first, then use F.embedding_bag to sum them efficiently.</p>

username: AndyTengWei
cooked: <p>Do you mean Attention in DL model?</p>