Retrieveing indices using nn.Embedding

username: Octocodist
cooked: <p>Hello,<br>
I am currently working on building a <strong>VAE for text</strong>.</p>
<p>My problem is, that I don`t understand, how to <strong>convert my embedded vectors back to tokens</strong> in the Decoder using the same embedding weights/ vocabulary as in the Encoder.</p>
<p>Some open questions I have:<br>
Do Encoder and Decoder need to use the same weights of nn.Embedding, i.e. the same mapping of tokens → embedding. ( Current assumption: Yes)<br>
Do I even need to transform the embeddings to tokens in order to get a good reconstruction loss? ( Current assumption: Yes)<br>
Do I need the mapping anyways if I want to generate new text? ( Current assumption: Yes)</p>
<p>Let me walk you through the current situation:<br>
First I use a tokenizer which produces a tensor of shape [batch_size, sentence_length] full of word tokens.<br>
Now the nn.Embedding(…) layer embedds each token, i.e. transforms the tensor to [batch_size, sent_length, emd_dim].</p>
<p>(… then other stuff happens)</p>
<p>In the decoder we finally get to the point where we have a tensor of the same dimension, i.e. [batch_size, sent_length, emd_dim] and we want convert this to tokens.</p>
<p><strong>Is there any smart way of doing this?</strong></p>
<p><strong>my naive/ brute force approach would be:</strong></p>
<p><em>For each element of the input:</em> ( each embedded vector[i,j,:])</p>
<p><strong>We want to iterate through all the embedding weights</strong></p>
<p>(the embedding of the entire vocabulary from nn.Embedding)</p>
<p><strong>and find the matching embedding vector.</strong></p>
<p>(if it exists, due to reconstruction error, i suppose there will be no exact match, hence we would need to compare to all the vectors in the nn.Embedding layer to find the one with minimal distance)</p>
<p><strong>From this vector, we can take the index and use it as a token.</strong><br>
(only if the ordering in nn.Embedding is the same as in the tokenizer )</p>
<p><strong>If the ordering is not the same, I would need to reembed each word from vocab and store the result in a table in order to lookup the results.</strong></p>
<p><strong>The way I implemented this approach, should be way too complicated, as the</strong> <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html" rel="noopener nofollow ugc">nn.Embedding site</a>  <strong>says that nn.Embedding is a:</strong></p>
<blockquote>
<p>A simple lookup table that stores embeddings of a fixed dictionary and size.</p>
</blockquote>
<blockquote>
<p>This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.</p>
</blockquote>
<p>TL,DR: nn.Embedding transforms tokens/indicies → embeddings I want to transform embeddings → tokens/indicies, using the same nn.Embedding as defined in the model in another layer.</p>
<p>Any help is greatly appreciated!</p>
<p>PS: This is my first post here so any feedback concerning layout ect. is also welcome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: vdw
cooked: <p>I have (hopefully) working <a href="https://github.com/chrisvdweth/ml-toolkit/tree/master/pytorch/models/text/autoencoder" rel="noopener nofollow ugc">implementations of AE and VAE based on a CNN and an RNN</a> for text. I did this long time ago, so I cannot really follow every single line right now. But maybe it gives you ideas.</p>
<p>Note that making an AE an VAE makes learning much more difficult. This seems to be a general observation and I would agree. Both AE works just fine, but both VAE – although in terms of code the changes a very minor – need much longer training to show results.</p>
<p>I hope that helps.</p>