Training with and without pretrained embeddings

username: Rohan_Kumar
cooked: <p>So I was trying to move away from torchtext and create my own custom dataloader, which i think I did. I tested it out with the IMDb dataset for sentiment analysis, but I forgot to add pretrained embeddings. So I have two questions for -<br>
1-  nn.Embeddings basically creates a 2d matrix of random weights, if pretrained is not provided. And starts training them via grad descent, is this right? If so does this mean there are additional vocab_size*emb_dim parameters to train?<br>
2- which brings to my follow up question, does the pretrained embeddings only function is to speed up training (by whatever factor, probably huge tho), I had the following graph of loss vs iterations without pretrained embeddings for 5 epochs. Do you think there might be any other issues in my code except embeddings? <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9ce5dac65cdaaf9e43b83be95e53ba2ef7e0accd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9ce5dac65cdaaf9e43b83be95e53ba2ef7e0accd" title="received_660986764424257"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9ce5dac65cdaaf9e43b83be95e53ba2ef7e0accd_2_690x388.jpeg" alt="received_660986764424257" data-base62-sha1="mnYWGDErERJvpkcxc3HmreSDeih" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9ce5dac65cdaaf9e43b83be95e53ba2ef7e0accd_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9ce5dac65cdaaf9e43b83be95e53ba2ef7e0accd_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9ce5dac65cdaaf9e43b83be95e53ba2ef7e0accd_2_1380x776.jpeg 2x" data-dominant-color="81898A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">received_660986764424257</span><span class="informations">1920×1080 189 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Abhilash_Srivastava
cooked: <aside class="quote no-group" data-username="Rohan_Kumar" data-post="1" data-topic="57533">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohan_kumar/48/10228_2.png" class="avatar"> Rohan_Kumar:</div>
<blockquote>
<p>1- nn.Embeddings basically creates a 2d matrix of random weights, if pretrained is not provided. And starts training them via grad descent, is this right? If so does this mean there are additional vocab_size*emb_dim parameters to train?</p>
</blockquote>
</aside>
<p>nn.Embeddings create a 2d matrix. But, you can choose the weight initialization (eg: random or something else). Yes, if you choose to train these embeddings, there are additional vocab_size*emb_dim parameters to train.</p>
<aside class="quote no-group" data-username="Rohan_Kumar" data-post="1" data-topic="57533">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohan_kumar/48/10228_2.png" class="avatar"> Rohan_Kumar:</div>
<blockquote>
<p>2- which brings to my follow up question, does the pretrained embeddings only function is to speed up training (by whatever factor, probably huge tho), I had the following graph of loss vs iterations without pretrained embeddings for 5 epochs. Do you think there might be any other issues in my code except embeddings?</p>
</blockquote>
</aside>
<p>Pre-trained embeddings not only reduce the number of parameters to train (hence reducing the train time), but also bring the “knowledge” of words (eg: word2vec) and context (eg: BERT embeddings). Hence can be directly used for downstream tasks.<br>
Clearly, your model is not learning much (loss not decreasing with epochs). It’s hard to figure out the reason just on the basis of this plot. Better use average loss (for each epoch) to getter a better sense of loss and investigate further.</p>

username: vdw
cooked: <p>Just to add to <a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a> post: Even if you used pretrained word embeddings you can decide to keep the fixed or to (further) train them. Keeping them fixed obviously increases training speed. Training them, however, might make them more adapt to our context, potentially yielding a higher accuracy.</p>
<p>Language is extremely multifaceted and different embedding approaches make different assumptions to map words into a vector space – that’s why we have so many embedding solutions in the first place. For example, the Skip-gram model behind Word2Vec considers to words/vectors similar, if they are used in the same context. Under this assumption, “ugly” and “beautiful” are rather close in the embedding space since words words are used to describe appearances. Thus, Word2Vec doesn’t directly capture the notion of polarity what is important for, e.g., sentiment analysis.</p>
<p>In short, even when using pretrained embeddings, try both: keeping them fixed vs. training them, and see what comes out of it.</p>
<p>Regarding your plot – although that’s pure anecdotal speculation – I don’t think that this is an embedding problem. Pretrained embeddings typically improve things but are now completely magic either. What about the first test one can do: Can you overtrain the network in a small sample of your dataset?</p>

username: Rohan_Kumar
cooked: <p>So i now i have the plot of average loss per epoch, and i have sampled 100 positive and 100 negative data points (total 200 out of 25000 texts) and ran the model for 50 epochs. this is the loss vs epoch i am getting. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b66c69377564b9014644304632d0864dd0854f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/4b66c69377564b9014644304632d0864dd0854f0" title="Screenshot_2019-10-06%20Google%20Colaboratory"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b66c69377564b9014644304632d0864dd0854f0_2_689x338.png" alt="Screenshot_2019-10-06%20Google%20Colaboratory" data-base62-sha1="aL20nKCcuV9wpGc4bnaH7nx5Q7C" width="689" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b66c69377564b9014644304632d0864dd0854f0_2_689x338.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b66c69377564b9014644304632d0864dd0854f0_2_1033x507.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b66c69377564b9014644304632d0864dd0854f0.png 2x" data-dominant-color="F2F6F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2019-10-06%20Google%20Colaboratory</span><span class="informations">1212×594 61.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have also included the model that i wrote -</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, vocab_size, emb_dim, hid_dim, dropout = 0.5):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, emb_dim)
        self.rnn = nn.GRU(emb_dim, hid_dim)
        self.fc = nn.Linear(hid_dim, 1)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, src):
#         print(src.size())
        src = src.permute(1,0)
#         print(src.size())
        embedded = self.embedding(src)
#         print(embedded.size())
        output, hidden = self.rnn(embedded)
#         print(output[-1].size())
        out = self.fc(self.dropout(output[-1]))
#         print(out.size())
        return F.softmax(out, dim=1)
</code></pre>
<p>again all this is done without pre-trained embeddings, i will do this with pre-trained also and upload the results.</p>

username: vdw
cooked: <p>According to <code>self.fc = nn.Linear(hid_dim, 1)</code> you only have one output. So I assume you’re doing regression and not classification. However, use a Softmax layer in the end. Softmax is usually used for classification to “generate” a probablity distribution over all classes. While I’ve never tried it, Softmax of just one output should always be 1.0. That might explain why your network is not learning anything? Try it with out Softmax.</p>

username: Rohan_Kumar
cooked: <p>I removed softmax from the code, and now the code looks like -</p>
<pre><code class="lang-auto">.....
return out
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">return F.softmax(out, dim = 1)
</code></pre>
<p>I trained it for 200 epochs, without pretrained embeddings. This is the loss vs epoch plot i got. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f980e0bf141c7042e4a400bac408f77aa1897d7.png" data-download-href="https://discuss.pytorch.org/uploads/default/1f980e0bf141c7042e4a400bac408f77aa1897d7" title="Screenshot_2019-10-06%20Google%20Colaboratory(1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f980e0bf141c7042e4a400bac408f77aa1897d7_2_690x348.png" alt="Screenshot_2019-10-06%20Google%20Colaboratory(1)" data-base62-sha1="4vuzOrKsYIGyLn0Qg4Qsan1teV9" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f980e0bf141c7042e4a400bac408f77aa1897d7_2_690x348.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f980e0bf141c7042e4a400bac408f77aa1897d7_2_1035x522.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f980e0bf141c7042e4a400bac408f77aa1897d7.png 2x" data-dominant-color="F3F7FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2019-10-06%20Google%20Colaboratory(1)</span><span class="informations">1177×594 73.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I dont understand, why is the loss increasing now? even without the the pretrained embeddings it should decrease albeit slowly.</p>

username: Rohan_Kumar
cooked: <p>I change my learning rate from 1e-3 to 1e-5, and trained it again for 200 epochs, I am getting a better plot now,<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/28071613f4573a7b9e2c01f0b8a735aa3753e0cf.png" data-download-href="https://discuss.pytorch.org/uploads/default/28071613f4573a7b9e2c01f0b8a735aa3753e0cf" title="Screenshot_2019-10-06%20Google%20Colaboratory(2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28071613f4573a7b9e2c01f0b8a735aa3753e0cf_2_690x347.png" alt="Screenshot_2019-10-06%20Google%20Colaboratory(2)" data-base62-sha1="5I6hqfgqElOU8nEMkQ4MgP7yyu3" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28071613f4573a7b9e2c01f0b8a735aa3753e0cf_2_690x347.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28071613f4573a7b9e2c01f0b8a735aa3753e0cf_2_1035x520.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/8/28071613f4573a7b9e2c01f0b8a735aa3753e0cf.png 2x" data-dominant-color="F3F7FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2019-10-06%20Google%20Colaboratory(2)</span><span class="informations">1197×602 75.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The loss is decreasing, till i guess 120-130 epochs? and then it kinda starts increasing, any possible explanation for this?</p>

username: Abhilash_Srivastava
cooked: <p>A couple of things:</p>
<ul>
<li>What is your loss function? For the sentiment analysis task should the sentiment fall in a range [0,1]? If yes, you should be using <code>nn.BCEWithLogitsLoss</code>.</li>
<li>If the same dataset is used across all the epochs, the train loss value should go down (or saturate) but never increase.<br>
Share your training code for better understanding.</li>
</ul>

username: Rohan_Kumar
cooked: <p>I am using BCEwithlogitsloss loss function, and this is the trainer function i defined.</p>
<pre><code class="lang-auto">def trainer(train_data, epochs):
  globaliter = 0
  x = []
  y = []
  for epoch in tqdm(range(epochs)):  
    epoch_loss = 0 
    epoch_acc = 0
    epoch_size = 0
    for batch in tqdm(train_data):
        globaliter += 1
        src = Variable(batch['text']).cuda()
        out = model(src)
        out = out.squeeze()
        trg = Variable(batch['sentiment']).cuda()
        loss = criterion(out, trg)
        loss.backward()    
        optimizer.step()
        epoch_loss += loss.item()
        acc = binary_accuracy(out, trg)
        epoch_acc += acc.item()
    y.append(epoch_loss/len(train_data))
    x.append(epoch+1)
  return x,y 
</code></pre>
<p>I will try and see what happens if i decrease my lr even more.</p>

username: Abhilash_Srivastava
cooked: <p>Did you miss using <code>optimizer.zero_grad()</code> before <code>loss.backward()</code> ?<br>
This can also be the reason why the loss curve is not smooth.</p>

username: Rohan_Kumar
cooked: <p>I added optimizer.zero_grad now the graph is so much more worse.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7562f43364db0fef2b7e5f24774d60e8a2d70a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/f7562f43364db0fef2b7e5f24774d60e8a2d70a4" title="Screenshot_2019-10-08%20Google%20Colaboratory"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7562f43364db0fef2b7e5f24774d60e8a2d70a4_2_690x345.png" alt="Screenshot_2019-10-08%20Google%20Colaboratory" data-base62-sha1="zi2Agl99S5vPxJO7n0GLIkmlVFq" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7562f43364db0fef2b7e5f24774d60e8a2d70a4_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7562f43364db0fef2b7e5f24774d60e8a2d70a4_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7562f43364db0fef2b7e5f24774d60e8a2d70a4.png 2x" data-dominant-color="EFF4F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2019-10-08%20Google%20Colaboratory</span><span class="informations">1179×590 88.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Rohan_Kumar
cooked: <p>I added return torch.sigmoid(out) inplace of return out and this is the graph i am getting. which seems a lot better. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c032d7687bbb8451cacd67abb2b1ae7211da1321.png" data-download-href="https://discuss.pytorch.org/uploads/default/c032d7687bbb8451cacd67abb2b1ae7211da1321" title="Screenshot_2019-10-08%20Google%20Colaboratory(2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c032d7687bbb8451cacd67abb2b1ae7211da1321_2_690x347.png" alt="Screenshot_2019-10-08%20Google%20Colaboratory(2)" data-base62-sha1="rqgBizhoGOUWHtPYjCiB2UL93uF" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c032d7687bbb8451cacd67abb2b1ae7211da1321_2_690x347.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c032d7687bbb8451cacd67abb2b1ae7211da1321_2_1035x520.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c032d7687bbb8451cacd67abb2b1ae7211da1321.png 2x" data-dominant-color="FAFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2019-10-08%20Google%20Colaboratory(2)</span><span class="informations">1188×599 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Abhilash_Srivastava
cooked: <p>Cool! What does your complete code look like now?<br>
I would still want the train loss to go close to zero (over-fit for a small sample).</p>