Word2Vec as input to lstm

username: chrome
cooked: <p>Hi all. I am new to Pytorch and wanted your help. I have created a word2vec model of a corpus using gensim w2v function. Now I want to feed this model to a Bidirectional lstm. How do I proceed with that? And what exactly is nn.embedding layer? Do I need it?</p>

username: vdw
cooked: <p>You essentially always need an embedding layer to map words to vector representations - the network doesn’t understand words, only (vectors/tensors) of numbers. Word embeddings such as Word2Vec or GloVe ensure that these vector representations have already a semantic meaning before ever training the network.</p>
<p>When creating an LSTM network the first layer is usually something like:</p>
<pre><code>self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
</code></pre>
<p>You can then, before training, set the weights of embedding layer to yout word vectors like that:</p>
<pre><code># The shape of the embedding_matrix must be (vocab_size, embedding_dim)
model.word_embeddings.weight.data.copy_(torch.from_numpy(embeddings_matrix))

# Make sure that the weights in the embedding layer are not updated
model.word_embeddings.weight.requires_grad=False
</code></pre>
<p>The main task is to create <code>embeddings_matrix</code> correctly. In my code, I use GloVe word vectors I load from a file. The following snippet just shows the rough idea since all this stuff is custom code.</p>
<pre><code>word_vector_loader = WordVectorLoader() #
word_vector_loader.load_glove('glove.6B.100d.txt') # embedding_dim=100
embeddings_matrix = word_vector_loader.generate_embedding_matrix(vectorizer.vocabulary.word_to_index, 100, max_idx)
</code></pre>
<p>Check out <code>torchtext</code> which might make this all much easier. At least it provides you with pretrained word vectors. If you insist to use your own, you probably have to do prepare them to serve as weights for the embeddings layer yourself.</p>
<p>It’s a bit difficult to be more helpful without seeing any code; I just copied the snippets from one of my projects.</p>

username: chrome
cooked: <p>Thank you Chris for the help especially the requires_grad part.<br>
So I have made some changes and these are the steps I followed:<br>
1: <code>model.save('w2v.model')</code>  # which persists the word2vec model I created using gensim<br>
2:  <code>model = Word2Vec.load('w2v.model')</code> # loading the model<br>
3:</p>
<pre><code class="lang-auto"> weights = torch.FloatTensor(model.wv.vectors)
embedding = nn.Embedding.from_pretrained(weights)
</code></pre>
<p>Does these steps seem correct(I haven’t added the requires_grad yet)? The w2v dimension is 200</p>

username: vdw
cooked: <p>I cannot test the code but it looks alright.</p>
<p>You only have to make sure that the input sequences match the embedding. For example, if you convert the sentence “i go to work every day” into the input sequence <code>[4, 24, 8, 120, 53, 78, 0, 0, 0, 0]</code> with 0 representing padding, so that 4 represents “I”, 24 represents “go” and so on…then that must match in the embedding matrix, i.e., the forth word vector is the one for “i”. See this <a href="https://discuss.pytorch.org/t/aligning-torchtext-vocab-index-to-loaded-embedding-pre-trained-weights/">other post</a> discussing that issue.</p>

username: JustAGuysInThailand
cooked: <p>Hi Chris,<br>
I follow your instruction but I got a problem… I didn’t find any solution. I hope that you can help me.</p>
<p>So basically the code for me is like this :</p>
<blockquote>
<p>w2v_model = gensim.models.Word2Vec.load(‘./model/word2vec.model’)<br>
w2v_weights = w2v_model.wv.vectors</p>
</blockquote>
<p>so the weight is shape (81505,100 )</p>
<p>so I put my model like this :</p>
<blockquote>
<p>class LSTM(nn.Module):<br>
“”“docstring for LSTM”“”<br>
def <strong>init</strong>(self):<br>
super(LSTM, self).<strong>init</strong>()<br>
self.word_embeddings = nn.Embedding(81505, 100)<br>
self.lstm = nn.LSTM(500, 300,2,bidirectional = True)<br>
self.dropout = nn.Dropout(0.5)<br>
self.dense = nn.Linear(200, 5)<br>
self.act = nn.ReLU()</p>
<p>def forward(self,x):<br>
text_emb = self.word_embeddings(x)<br>
lstm_out, lstm_hidden = self.lstm(text_emb)<br>
lstm_out = lstm_out[:,-1,:]<br>
lstm_out = self.act(lstm_out)<br>
drop_out = self.dropout(lstm_out)<br>
output = self.dense(drop_out)<br>
return output</p>
</blockquote>
<p>But then after begin the training I got this error :</p>
<blockquote>
<pre><code>return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
</code></pre>
<p>IndexError: index out of range in self</p>
</blockquote>
<p>Sorry if my question is too obvious. I’m really new with NLP.</p>

username: vdw
cooked: <p>Hm, I’m not sure how to interpret the error. It seem to come from a torch package? If your aim is to get your code running, I would try the following to simplify the model – makes debugging easier:</p>
<ul>
<li>
<p>Name your class <code>MyLSTM</code>. It’s probably not an issue but there’s not need to risk naming problems :).</p>
</li>
<li>
<p>Use just one layer and unidirectional: <code>self.lstm = nn.LSTM(500, 300,1,bidirectional=False)</code>. Otherwise <code>lstm_out = lstm_out[:,-1,:]</code> might yield unexpected results</p>
</li>
<li>
<p>Remove (comment out) <code>self.dropout</code> and <code>self.act</code>. <code>nn.LSTM</code> already use activation functions, an Dropout is not needed to get it working, “just” to improve the results.</p>
</li>
<li>
<p>Since you don’t show the code, I’m not sure if you set the pretrained word vectors. Anyway, again, they are not needed to get a working model for a start</p>
</li>
<li>
<p>The <code>hidden_dim</code> of you <code>nn.LSTM</code> is 300, but you define <code>nn.Linear(200, 5)</code>. How do you get from 300 dimensions to 200 for the linear layer?</p>
</li>
<li>
<p>Can you change your <code>forward()</code> method to</p>
</li>
</ul>
<pre><code class="lang-auto">def forward(self,x):
    print(x.shape)
    text_emb = self.word_embeddings(x)
    print(text_emb.shape)
    lstm_out, lstm_hidden = self.lstm(text_emb)
    lstm_out = lstm_out[:,-1,:]
    print(lstm_out.shape)
    output = self.dense(drop_out)
    return output
</code></pre>

username: JustAGuysInThailand
cooked: <p>Thank you for your answer.</p>
<p>I think maybe I did use the word2vec weight to convert my words before feeding to the LSTM. So I believe that my input already been embedding? So I don’t need the embedding layer anymore. I’m not so sure if that is correct.</p>
<p>Here is my code to convert data before input to the model :</p>
<blockquote>
<p>for sentence in txt_sequence:<br>
temp = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for word in sentence :<br>
temp.append(w2v_model.wv[word]) ## this will extract the vector of the word from weight of w2v<br>
x.append(temp)</p>
</blockquote>
<p>After that, I pad my x with padding-zero provide from Keras. So I did remove the embedding layers and everything work. But I’m not sure is it the most efficient way.</p>